Nov 04 00:57:49.896847 ip-10-0-140-241 systemd[1]: Starting Kubernetes Kubelet... Nov 04 00:57:50.303578 ip-10-0-140-241 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:57:50.303578 ip-10-0-140-241 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 00:57:50.303578 ip-10-0-140-241 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:57:50.303578 ip-10-0-140-241 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:57:50.308573 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308487 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:50.308614 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308587 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:50.308614 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308593 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:50.308614 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308596 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:50.308614 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308599 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:50.308614 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308602 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:50.308614 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308605 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:50.308614 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308609 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:50.308614 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308611 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:50.308614 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308614 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:50.308614 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308617 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:50.308614 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308620 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:50.308874 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308623 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:50.308874 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308625 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:50.308874 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308627 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:50.308874 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308630 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:50.308874 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308633 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:50.308874 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308635 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:50.308874 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308638 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:50.308874 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308645 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:50.308874 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308649 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:50.308874 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308652 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:50.308874 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308654 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:50.308874 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308657 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:50.308874 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308659 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:50.308874 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308662 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:50.308874 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308665 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:50.308874 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308667 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:50.308874 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308670 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:50.308874 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308672 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:50.308874 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308675 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:50.308874 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308677 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:50.309330 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308682 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:50.309330 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308686 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:50.309330 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308689 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:50.309330 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308692 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:50.309330 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308694 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:50.309330 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308697 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:50.309330 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308699 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:50.309330 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308702 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:50.309330 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308704 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:50.309330 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308706 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:50.309330 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308709 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:50.309330 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308712 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:50.309330 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308715 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:50.309330 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308718 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:50.309330 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308721 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:50.309330 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308724 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:50.309330 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308727 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:50.309330 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308730 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:50.309330 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308732 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:50.309330 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308735 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:50.309826 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308738 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:50.309826 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308740 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:50.309826 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308743 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:50.309826 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308745 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:50.309826 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308748 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:50.309826 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308751 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:50.309826 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308753 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:50.309826 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308756 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:50.309826 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308758 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:50.309826 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308760 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:50.309826 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308763 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:50.309826 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308765 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:50.309826 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308768 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:50.309826 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308770 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:50.309826 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308773 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:50.309826 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308775 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:50.309826 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308778 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:50.309826 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308780 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:50.309826 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308782 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:50.309826 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308785 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:50.310322 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308787 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:50.310322 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308790 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:50.310322 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308792 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:50.310322 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308796 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:50.310322 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308800 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:50.310322 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308804 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:50.310322 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308807 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:50.310322 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308810 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:50.310322 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308812 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:50.310322 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308815 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:50.310322 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308817 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:50.310322 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308820 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:50.310322 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308823 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:50.310322 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308825 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:50.310322 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308828 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:50.310322 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308830 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:50.310322 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308834 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:50.310322 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308836 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:50.310322 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308839 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:50.310785 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308841 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:50.310785 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308843 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:50.310785 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308846 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:50.310785 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308848 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:50.310785 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308851 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:50.310785 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308854 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:50.310785 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308858 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:50.310785 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308862 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:50.310785 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308866 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:50.310785 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308869 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:50.310785 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308871 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:50.310785 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308874 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:50.310785 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308876 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:50.310785 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308879 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:50.310785 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.308882 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:50.310785 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309666 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:50.310785 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309673 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:50.310785 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309676 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:50.310785 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309681 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:50.310785 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309683 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:50.311257 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309686 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:50.311257 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309689 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:50.311257 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309692 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:50.311257 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309695 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:50.311257 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309698 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:50.311257 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309701 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:50.311257 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309704 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:50.311257 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309706 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:50.311257 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309709 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:50.311257 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309711 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:50.311257 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309714 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:50.311257 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309716 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:50.311257 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309719 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:50.311257 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309721 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:50.311257 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309723 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:50.311257 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309726 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:50.311257 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309728 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:50.311257 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309731 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:50.311257 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309734 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:50.311257 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309736 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:50.311783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309739 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:50.311783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309741 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:50.311783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309744 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:50.311783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309746 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:50.311783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309749 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:50.311783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309751 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:50.311783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309754 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:50.311783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309756 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:50.311783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309759 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:50.311783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309762 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:50.311783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309764 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:50.311783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309767 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:50.311783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309770 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:50.311783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309773 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:50.311783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309775 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:50.311783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309779 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:50.311783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309784 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:50.311783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309787 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:50.311783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309790 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:50.311783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309793 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:50.312384 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309795 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:50.312384 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309798 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:50.312384 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309800 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:50.312384 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309803 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:50.312384 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309806 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:50.312384 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309808 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:50.312384 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309811 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:50.312384 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309813 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:50.312384 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309816 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:50.312384 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309818 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:50.312384 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309821 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:50.312384 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309823 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:50.312384 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309825 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:50.312384 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309829 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:50.312384 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309832 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:50.312384 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309834 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:50.312384 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309837 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:50.312384 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309840 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:50.312384 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309842 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:50.312384 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309845 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:50.312888 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309847 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:50.312888 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309850 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:50.312888 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309853 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:50.312888 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309856 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:50.312888 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309858 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:50.312888 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309861 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:50.312888 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309863 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:50.312888 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309866 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:50.312888 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309868 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:50.312888 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309877 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:50.312888 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309880 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:50.312888 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309883 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:50.312888 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309885 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:50.312888 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309887 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:50.312888 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309890 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:50.312888 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309892 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:50.312888 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309895 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:50.312888 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309898 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:50.312888 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309901 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:50.313338 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309903 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:50.313338 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309906 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:50.313338 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309908 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:50.313338 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309911 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:50.313338 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309919 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:50.313338 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309922 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:50.313338 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309924 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:50.313338 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309927 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:50.313338 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309929 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:50.313338 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309932 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:50.313338 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309935 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:50.313338 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309937 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:50.313338 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309940 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:50.313338 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309943 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:50.313338 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309948 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:50.313338 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309950 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:50.313338 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309953 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:50.313338 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309956 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:50.313783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309958 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:50.313783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309961 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:50.313783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309964 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:50.313783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.309966 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:50.313783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310474 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:50.313783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310479 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:50.313783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310482 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:50.313783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310485 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:50.313783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310488 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:50.313783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310491 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:50.313783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310493 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:50.313783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310496 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:50.313783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310499 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:50.313783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310501 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:50.313783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310504 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:50.313783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310507 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:50.313783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310509 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:50.313783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310512 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:50.313783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310514 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:50.313783 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310517 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:50.314258 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310519 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:50.314258 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310521 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:50.314258 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310524 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:50.314258 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310526 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:50.314258 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310529 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:50.314258 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310531 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:50.314258 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310534 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:50.314258 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310536 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:50.314258 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310539 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:50.314258 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310541 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:50.314258 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310544 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:50.314258 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310546 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:50.314258 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310548 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:50.314258 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310551 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:50.314258 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310553 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:50.314258 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310556 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:50.314258 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310558 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:50.314258 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310583 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:50.314258 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310586 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:50.314717 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310588 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:50.314717 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310591 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:50.314717 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310593 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:50.314717 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310596 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:50.314717 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310599 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:50.314717 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310601 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:50.314717 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310605 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:50.314717 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310607 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:50.314717 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310610 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:50.314717 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310612 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:50.314717 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310615 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:50.314717 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310618 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:50.314717 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310620 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:50.314717 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310622 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:50.314717 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310625 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:50.314717 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310627 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:50.314717 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310630 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:50.314717 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310633 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:50.314717 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310635 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:50.315153 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310638 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:50.315153 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310640 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:50.315153 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310643 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:50.315153 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310645 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:50.315153 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310648 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:50.315153 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310650 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:50.315153 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310653 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:50.315153 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310655 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:50.315153 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310658 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:50.315153 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310660 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:50.315153 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310663 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:50.315153 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310666 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:50.315153 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310674 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:50.315153 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310677 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:50.315153 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310679 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:50.315153 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310682 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:50.315153 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310685 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:50.315153 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310687 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:50.315153 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310690 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:50.315153 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310692 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:50.315631 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310695 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:50.315631 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310697 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:50.315631 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310699 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:50.315631 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310702 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:50.315631 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310704 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:50.315631 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310707 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:50.315631 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310711 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:50.315631 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310715 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:50.315631 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310718 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:50.315631 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310720 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:50.315631 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310723 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:50.315631 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310725 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:50.315631 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310728 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:50.315631 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310730 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:50.315631 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310733 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:50.315631 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310735 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:50.315631 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310737 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:50.315631 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310740 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:50.315631 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310742 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:50.316096 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310745 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:50.316096 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310747 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:50.316096 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310750 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:50.316096 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310753 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:50.316096 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310755 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:50.316096 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310758 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:50.316096 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310765 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:50.316096 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310768 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:50.316096 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310771 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:50.316096 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310773 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:50.316096 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310775 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:50.316096 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310778 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:50.316096 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.310780 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:50.316096 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311798 2579 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 00:57:50.316096 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311808 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 00:57:50.316096 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311818 2579 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 00:57:50.316096 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311823 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 00:57:50.316096 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311828 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 00:57:50.316096 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311831 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 00:57:50.316096 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311836 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 00:57:50.316096 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311840 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311843 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311846 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311850 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311853 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311856 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311858 2579 flags.go:64] FLAG: --cgroup-root="" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311861 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311870 2579 flags.go:64] FLAG: --client-ca-file="" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311873 2579 flags.go:64] FLAG: --cloud-provider="external" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311875 2579 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311881 2579 flags.go:64] FLAG: --cluster-domain="" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311884 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311890 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311893 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311896 2579 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311901 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311904 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311907 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311916 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311920 2579 flags.go:64] FLAG: --contention-profiling="false" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311922 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311925 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311929 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 00:57:50.316588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311931 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311936 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311939 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311942 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311944 2579 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311947 2579 flags.go:64] FLAG: --enable-server="true" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311951 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311959 2579 flags.go:64] FLAG: --event-burst="100" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311962 2579 flags.go:64] FLAG: --event-qps="50" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311965 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311968 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311971 2579 flags.go:64] FLAG: --eviction-hard="" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311976 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311979 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311982 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311985 2579 flags.go:64] FLAG: --eviction-soft="" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311988 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311990 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311993 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311996 2579 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.311999 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312004 2579 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312007 2579 flags.go:64] FLAG: --feature-gates="" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312011 2579 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312015 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 00:57:50.317196 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312018 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312021 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312024 2579 flags.go:64] FLAG: --healthz-port="10248" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312033 2579 flags.go:64] FLAG: --help="false" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312036 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312039 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312042 2579 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312045 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312048 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312052 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312054 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312057 2579 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312060 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312062 2579 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312065 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312068 2579 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312071 2579 flags.go:64] FLAG: --kube-reserved="" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312074 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312077 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312080 2579 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312082 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312085 2579 flags.go:64] FLAG: --lock-file="" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312088 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312091 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 00:57:50.317815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312094 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312099 2579 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312102 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312105 2579 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312110 2579 flags.go:64] FLAG: --logging-format="text" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312113 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312116 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312120 2579 flags.go:64] FLAG: --manifest-url="" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312123 2579 flags.go:64] FLAG: --manifest-url-header="" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312127 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312130 2579 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312133 2579 flags.go:64] FLAG: --max-pods="110" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312143 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312146 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312149 2579 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312152 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312155 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312157 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312160 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312167 2579 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312170 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312173 2579 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312176 2579 flags.go:64] FLAG: --pod-cidr="" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312179 2579 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312182 2579 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 00:57:50.318399 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312184 2579 flags.go:64] FLAG: --pods-per-core="0" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312187 2579 flags.go:64] FLAG: --port="10250" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312190 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312193 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0312e0f14a1f7850b" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312196 2579 flags.go:64] FLAG: --qos-reserved="" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312199 2579 flags.go:64] FLAG: --read-only-port="10255" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312202 2579 flags.go:64] FLAG: --register-node="true" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312205 2579 flags.go:64] FLAG: --register-with-taints="" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312208 2579 flags.go:64] FLAG: --registry-burst="10" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312211 2579 flags.go:64] FLAG: --registry-qps="5" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312214 2579 flags.go:64] FLAG: --reserved-cpus="" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312218 2579 flags.go:64] FLAG: --reserved-memory="" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312222 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312225 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312229 2579 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312232 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312235 2579 flags.go:64] FLAG: --runonce="false" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312238 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312241 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312244 2579 flags.go:64] FLAG: --seccomp-default="false" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312252 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312255 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312258 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312261 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312264 2579 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312267 2579 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 00:57:50.319008 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312270 2579 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312272 2579 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312276 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312278 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312281 2579 flags.go:64] FLAG: --system-cgroups="" Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312284 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312289 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312292 2579 flags.go:64] FLAG: --tls-cert-file="" Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312295 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312301 2579 flags.go:64] FLAG: --tls-min-version="" Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312303 2579 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312306 2579 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312309 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312312 2579 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312315 2579 flags.go:64] FLAG: --v="2" Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312319 2579 flags.go:64] FLAG: --version="false" Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312323 2579 flags.go:64] FLAG: --vmodule="" Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312328 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312337 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312524 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312530 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312532 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312535 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312538 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:50.319638 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312540 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:50.320201 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312542 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:50.320201 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312545 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:50.320201 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312556 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:50.320201 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312576 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:50.320201 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312581 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:50.320201 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312585 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:50.320201 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312589 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:50.320201 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312592 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:50.320201 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312595 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:50.320201 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312597 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:50.320201 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312600 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:50.320201 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312603 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:50.320201 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312605 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:50.320201 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312608 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:50.320201 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312610 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:50.320201 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312613 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:50.320201 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312615 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:50.320201 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312619 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:50.320201 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312623 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:50.320663 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312626 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:50.320663 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312630 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:50.320663 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312633 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:50.320663 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312635 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:50.320663 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312638 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:50.320663 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312642 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:50.320663 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312644 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:50.320663 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312646 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:50.320663 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312650 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:50.320663 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312653 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:50.320663 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312656 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:50.320663 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312658 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:50.320663 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312661 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:50.320663 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312663 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:50.320663 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312666 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:50.320663 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312668 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:50.320663 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312677 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:50.320663 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312680 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:50.320663 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312682 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:50.320663 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312685 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:50.321167 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312687 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:50.321167 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312690 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:50.321167 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312693 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:50.321167 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312695 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:50.321167 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312698 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:50.321167 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312700 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:50.321167 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312703 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:50.321167 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312705 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:50.321167 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312708 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:50.321167 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312710 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:50.321167 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312713 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:50.321167 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312715 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:50.321167 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312717 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:50.321167 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312720 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:50.321167 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312722 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:50.321167 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312724 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:50.321167 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312727 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:50.321167 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312730 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:50.321167 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312733 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:50.321167 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312735 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:50.321664 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312739 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:50.321664 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312742 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:50.321664 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312746 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:50.321664 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312749 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:50.321664 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312752 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:50.321664 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312755 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:50.321664 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312758 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:50.321664 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312761 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:50.321664 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312764 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:50.321664 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312772 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:50.321664 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312775 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:50.321664 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312777 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:50.321664 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312779 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:50.321664 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312782 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:50.321664 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312784 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:50.321664 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312787 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:50.321664 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312789 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:50.321664 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312792 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:50.321664 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312794 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:50.322111 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312796 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:50.322111 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312799 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:50.322111 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312801 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:50.322111 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312804 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:50.322111 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312806 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:50.322111 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312809 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:50.322111 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312811 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:50.322111 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312814 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:50.322111 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312816 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:50.322111 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312818 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:50.322111 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312822 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:50.322111 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312825 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:50.322111 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312827 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:50.322111 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312831 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:50.322111 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312833 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:50.322111 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312836 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:50.322111 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312838 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:50.322111 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312841 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:50.322111 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312843 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:50.322111 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312846 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:50.322675 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312848 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:50.322675 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312851 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:50.322675 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.312879 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:57:50.322675 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312927 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:50.322675 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312931 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:50.322675 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312934 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:50.322675 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312936 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:50.322675 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312939 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:50.322675 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312942 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:50.322675 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312944 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:50.322675 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312947 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:50.322675 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312949 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:50.322675 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312952 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:50.322675 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312954 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:50.322675 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312957 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:50.322675 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312959 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:50.323043 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312962 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:50.323043 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312964 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:50.323043 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312967 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:50.323043 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312969 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:50.323043 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312971 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:50.323043 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312974 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:50.323043 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312976 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:50.323043 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312980 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:50.323043 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312984 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:50.323043 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312989 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:50.323043 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312992 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:50.323043 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312995 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:50.323043 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.312998 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:50.323043 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313001 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:50.323043 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313004 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:50.323043 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313006 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:50.323043 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313009 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:50.323043 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313012 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:50.323043 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313014 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:50.323583 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313017 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:50.323583 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313024 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:50.323583 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313027 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:50.323583 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313030 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:50.323583 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313032 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:50.323583 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313035 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:50.323583 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313037 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:50.323583 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313039 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:50.323583 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313042 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:50.323583 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313044 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:50.323583 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313046 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:50.323583 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313049 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:50.323583 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313051 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:50.323583 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313054 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:50.323583 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313056 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:50.323583 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313059 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:50.323583 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313061 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:50.323583 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313064 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:50.323583 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313066 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:50.323583 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313068 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:50.324347 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313079 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:50.324347 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313082 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:50.324347 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313084 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:50.324347 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313088 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:50.324347 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313090 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:50.324347 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313093 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:50.324347 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313095 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:50.324347 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313098 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:50.324347 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313101 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:50.324347 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313103 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:50.324347 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313106 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:50.324347 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313108 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:50.324347 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313110 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:50.324347 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313113 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:50.324347 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313121 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:50.324347 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313124 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:50.324347 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313126 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:50.324347 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313128 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:50.324347 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313131 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:50.324818 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313133 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:50.324818 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313135 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:50.324818 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313138 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:50.324818 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313140 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:50.324818 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313143 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:50.324818 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313145 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:50.324818 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313148 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:50.324818 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313150 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:50.324818 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313152 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:50.324818 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313155 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:50.324818 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313157 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:50.324818 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313160 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:50.324818 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313163 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:50.324818 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313168 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:50.324818 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313171 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:50.324818 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313174 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:50.324818 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313176 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:50.324818 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313179 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:50.324818 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313181 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:50.325266 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313184 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:50.325266 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313186 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:50.325266 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313189 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:50.325266 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313191 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:50.325266 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313194 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:50.325266 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313196 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:50.325266 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313199 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:50.325266 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313201 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:50.325266 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313204 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:50.325266 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313211 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:50.325266 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313214 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:50.325266 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313217 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:50.325266 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313219 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:50.325266 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313223 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:50.325266 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313225 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:50.325266 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.313227 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:50.325266 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.320175 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 00:57:50.325266 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.320190 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 00:57:50.325266 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.320776 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 00:57:50.325266 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.320782 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 00:57:50.325762 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320848 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:50.325762 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320855 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:50.325762 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320858 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:50.325762 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320861 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:50.325762 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320865 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:50.325762 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320868 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:50.325762 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320871 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:50.325762 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320874 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:50.325762 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320877 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:50.325762 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320880 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:50.325762 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320882 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:50.325762 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320885 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:50.325762 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320887 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:50.325762 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320890 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:50.325762 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320892 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:50.325762 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320895 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:50.325762 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320897 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:50.325762 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320900 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:50.325762 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320903 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:50.325762 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320906 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:50.326226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320909 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:50.326226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320911 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:50.326226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320914 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:50.326226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320916 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:50.326226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320919 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:50.326226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320921 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:50.326226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320924 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:50.326226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320926 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:50.326226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320930 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:50.326226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320933 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:50.326226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320935 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:50.326226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320938 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:50.326226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320940 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:50.326226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320943 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:50.326226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320946 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:50.326226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320948 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:50.326226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320951 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:50.326226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320953 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:50.326226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320956 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:50.326226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320958 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:50.326798 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320961 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:50.326798 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320963 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:50.326798 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320967 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:50.326798 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320971 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:50.326798 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320974 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:50.326798 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320977 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:50.326798 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320979 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:50.326798 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320982 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:50.326798 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320985 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:50.326798 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320987 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:50.326798 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320990 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:50.326798 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320994 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:50.326798 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.320997 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:50.326798 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321001 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:50.326798 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321004 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:50.326798 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321007 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:50.326798 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321009 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:50.326798 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321011 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:50.327248 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321014 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:50.327248 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321016 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:50.327248 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321019 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:50.327248 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321022 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:50.327248 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321024 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:50.327248 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321027 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:50.327248 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321029 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:50.327248 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321032 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:50.327248 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321034 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:50.327248 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321037 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:50.327248 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321039 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:50.327248 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321041 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:50.327248 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321044 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:50.327248 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321047 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:50.327248 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321049 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:50.327248 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321052 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:50.327248 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321054 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:50.327248 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321056 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:50.327248 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321059 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:50.327248 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321061 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:50.327734 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321063 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:50.327734 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321066 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:50.327734 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321068 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:50.327734 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321070 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:50.327734 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321073 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:50.327734 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321075 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:50.327734 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321078 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:50.327734 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321080 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:50.327734 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321083 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:50.327734 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321086 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:50.327734 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321088 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:50.327734 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321091 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:50.327734 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321093 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:50.327734 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321096 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:50.327734 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321098 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:50.327734 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321101 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:50.327734 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321103 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:50.327734 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321106 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:50.327734 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321108 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:50.327734 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321111 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:50.328209 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321113 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:50.328209 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321116 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:50.328209 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321118 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:50.328209 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321121 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:50.328209 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321123 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:50.328209 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321125 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:50.328209 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321128 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:50.328209 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321130 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:50.328209 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.321158 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:57:50.328209 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321308 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:50.328209 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321313 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:50.328209 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321317 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:50.328209 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321321 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:50.328209 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321323 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:50.328209 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321326 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:50.328590 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321329 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:50.328590 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321332 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:50.328590 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321335 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:50.328590 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321337 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:50.328590 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321340 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:50.328590 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321343 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:50.328590 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321346 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:50.328590 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321348 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:50.328590 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321351 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:50.328590 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321354 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:50.328590 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321356 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:50.328590 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321359 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:50.328590 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321361 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:50.328590 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321364 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:50.328590 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321366 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:50.328590 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321368 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:50.328590 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321371 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:50.328590 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321373 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:50.328590 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321375 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:50.328590 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321378 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:50.329226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321380 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:50.329226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321382 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:50.329226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321385 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:50.329226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321388 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:50.329226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321390 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:50.329226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321393 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:50.329226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321395 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:50.329226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321397 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:50.329226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321400 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:50.329226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321402 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:50.329226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321404 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:50.329226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321407 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:50.329226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321409 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:50.329226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321412 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:50.329226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321414 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:50.329226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321416 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:50.329226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321419 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:50.329226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321421 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:50.329226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321423 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:50.329226 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321426 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:50.329722 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321429 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:50.329722 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321440 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:50.329722 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321444 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:50.329722 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321447 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:50.329722 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321449 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:50.329722 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321452 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:50.329722 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321454 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:50.329722 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321457 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:50.329722 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321459 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:50.329722 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321462 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:50.329722 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321464 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:50.329722 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321467 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:50.329722 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321469 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:50.329722 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321472 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:50.329722 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321474 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:50.329722 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321477 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:50.329722 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321479 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:50.329722 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321481 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:50.329722 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321484 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:50.330175 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321486 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:50.330175 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321489 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:50.330175 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321491 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:50.330175 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321494 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:50.330175 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321497 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:50.330175 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321500 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:50.330175 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321503 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:50.330175 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321505 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:50.330175 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321508 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:50.330175 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321510 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:50.330175 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321512 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:50.330175 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321515 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:50.330175 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321517 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:50.330175 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321521 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:50.330175 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321523 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:50.330175 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321526 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:50.330175 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321528 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:50.330175 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321531 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:50.330175 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321533 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:50.330666 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321536 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:50.330666 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321538 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:50.330666 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321540 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:50.330666 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321543 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:50.330666 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321545 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:50.330666 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321548 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:50.330666 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321550 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:50.330666 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321553 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:50.330666 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321555 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:50.330666 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321558 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:50.330666 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321572 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:50.330666 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321575 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:50.330666 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321578 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:50.330666 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321580 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:50.330666 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321583 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:50.330666 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321585 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:50.330666 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321588 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:50.330666 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321590 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:50.330666 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321593 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:50.330666 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321595 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:50.331121 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321598 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:50.331121 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321601 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:50.331121 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.321624 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:57:50.331121 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321657 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:50.331121 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321663 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:50.331121 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321666 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:50.331121 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321669 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:50.331121 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321672 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:50.331121 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321675 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:50.331121 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321678 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:50.331121 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321681 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:50.331121 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321683 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:50.331121 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321685 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:50.331121 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321688 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:50.331121 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321690 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:50.331121 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321692 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:50.331491 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321697 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:50.331491 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321700 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:50.331491 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321702 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:50.331491 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321705 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:50.331491 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321707 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:50.331491 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321710 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:50.331491 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321712 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:50.331491 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321715 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:50.331491 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321717 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:50.331491 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321720 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:50.331491 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321723 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:50.331491 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321725 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:50.331491 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321727 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:50.331491 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321730 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:50.331491 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321733 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:50.331491 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321735 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:50.331491 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321738 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:50.331491 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321740 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:50.331491 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321743 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:50.331940 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321747 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:50.331940 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321751 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:50.331940 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321754 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:50.331940 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321757 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:50.331940 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321760 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:50.331940 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321763 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:50.331940 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321765 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:50.331940 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321767 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:50.331940 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321770 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:50.331940 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321773 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:50.331940 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321775 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:50.331940 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321777 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:50.331940 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321780 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:50.331940 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321783 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:50.331940 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321786 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:50.331940 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321788 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:50.331940 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321791 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:50.331940 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321793 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:50.331940 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321796 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:50.332399 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321798 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:50.332399 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321801 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:50.332399 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321803 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:50.332399 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321806 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:50.332399 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321809 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:50.332399 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321811 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:50.332399 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321814 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:50.332399 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321816 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:50.332399 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321819 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:50.332399 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321822 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:50.332399 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321824 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:50.332399 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321827 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:50.332399 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321829 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:50.332399 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321832 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:50.332399 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321835 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:50.332399 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321838 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:50.332399 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321840 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:50.332399 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321843 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:50.332399 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321846 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:50.332859 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321850 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:50.332859 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321853 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:50.332859 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321856 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:50.332859 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321859 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:50.332859 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321861 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:50.332859 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321864 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:50.332859 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321866 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:50.332859 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321869 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:50.332859 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321872 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:50.332859 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321875 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:50.332859 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321877 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:50.332859 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321879 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:50.332859 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321882 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:50.332859 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321885 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:50.332859 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321887 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:50.332859 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321890 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:50.332859 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321892 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:50.332859 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321894 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:50.332859 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321897 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:50.332859 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321899 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:50.333355 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321902 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:50.333355 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321904 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:50.333355 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321907 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:50.333355 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321909 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:50.333355 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321911 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:50.333355 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321914 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:50.333355 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321916 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:50.333355 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321921 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:50.333355 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321924 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:50.333355 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321927 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:50.333355 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321929 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:50.333355 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321931 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:50.333355 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321934 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:50.333355 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321937 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:50.333355 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321939 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:50.333355 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:50.321942 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:50.333355 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.321997 2579 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 00:57:50.333355 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.324215 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 04 00:57:50.333355 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.325303 2579 server.go:1014] "Starting client certificate rotation" Nov 04 00:57:50.333997 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.325408 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 00:57:50.333997 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.325455 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 00:57:50.333997 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.327461 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 00:57:50.333997 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.328574 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 00:57:50.341183 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.341167 2579 log.go:25] "Validated CRI v1 runtime API" Nov 04 00:57:50.344040 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.344022 2579 log.go:25] "Validated CRI v1 image API" Nov 04 00:57:50.344871 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.344847 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 00:57:50.348173 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.348154 2579 fs.go:135] Filesystem UUIDs: map[04f6d7dc-ee37-48f4-97f3-b2eb09cef2f0:/dev/nvme0n1p3 709d32f6-32ce-45ff-ad33-917e30cecb0c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 04 00:57:50.348231 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.348173 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 04 00:57:50.353094 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.352985 2579 manager.go:217] Machine: {Timestamp:2026-11-04 00:57:50.351857607 +0000 UTC m=+0.335395459 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097263 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2519dbc6858107bacce90bbc2fc5da SystemUUID:ec2519db-c685-8107-bacc-e90bbc2fc5da BootID:5a0d2f3f-4841-463d-849a-fc8f7d213cae Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ed:ff:7c:eb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ed:ff:7c:eb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:33:c3:e0:9d:f0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 00:57:50.353536 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.353525 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 00:57:50.353657 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.353645 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 00:57:50.356379 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.356354 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 00:57:50.356524 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.356381 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-241.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 00:57:50.356577 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.356526 2579 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 00:57:50.356577 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.356534 2579 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 00:57:50.356638 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.356597 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 00:57:50.356638 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.356619 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 00:57:50.357280 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.357270 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 00:57:50.357904 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.357887 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 00:57:50.361346 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.361334 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 00:57:50.361486 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.361476 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 00:57:50.361524 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.361514 2579 kubelet.go:493] "Attempting to sync node with API server" Nov 04 00:57:50.361550 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.361527 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 00:57:50.361550 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.361537 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 00:57:50.361550 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.361550 2579 kubelet.go:405] "Adding apiserver pod source" Nov 04 00:57:50.361662 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.361558 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 00:57:50.362124 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.362111 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 00:57:50.365954 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.365934 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 00:57:50.370300 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.370287 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 00:57:50.371102 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.371089 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 00:57:50.371154 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.371139 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 00:57:50.371324 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.371314 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 00:57:50.371354 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.371330 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 00:57:50.371354 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.371336 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 00:57:50.371354 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.371341 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 00:57:50.371354 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.371347 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 00:57:50.371354 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.371353 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 00:57:50.371886 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.371872 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 00:57:50.371917 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.371891 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 00:57:50.371917 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.371903 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 00:57:50.371917 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.371913 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 00:57:50.372040 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.371926 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 00:57:50.372040 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.371939 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 00:57:50.373131 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.373116 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 00:57:50.373165 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.373134 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 00:57:50.376879 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.376864 2579 server.go:1263] "Started kubelet" Nov 04 00:57:50.377004 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.376942 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 00:57:50.377173 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.377027 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 00:57:50.377240 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.377193 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 00:57:50.377725 ip-10-0-140-241 systemd[1]: Started Kubernetes Kubelet. Nov 04 00:57:50.378189 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.378159 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 00:57:50.379133 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.379120 2579 server.go:317] "Adding debug handlers to kubelet server" Nov 04 00:57:50.379809 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.379784 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pcck9" Nov 04 00:57:50.381823 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.381783 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 04 00:57:50.381823 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.381788 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-241.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 04 00:57:50.381939 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.381836 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-241.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 04 00:57:50.381939 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.381834 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 04 00:57:50.382728 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.381838 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4324b0c59 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.376832089 +0000 UTC m=+0.360369946,LastTimestamp:2026-11-04 00:57:50.376832089 +0000 UTC m=+0.360369946,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.383052 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.383036 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 00:57:50.383499 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.383476 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 00:57:50.383683 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.383667 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:50.383989 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.383972 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 00:57:50.384053 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.383965 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 00:57:50.384053 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.384018 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 00:57:50.384129 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.384091 2579 reconstruct.go:97] "Volume reconstruction finished" Nov 04 00:57:50.384129 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.384104 2579 reconciler.go:29] "Reconciler: start to sync state" Nov 04 00:57:50.386230 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.386208 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-241.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 04 00:57:50.386867 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.386844 2579 factory.go:153] Registering CRI-O factory Nov 04 00:57:50.386867 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.386860 2579 factory.go:223] Registration of the crio container factory successfully Nov 04 00:57:50.386992 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.386909 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 00:57:50.386992 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.386918 2579 factory.go:55] Registering systemd factory Nov 04 00:57:50.386992 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.386926 2579 factory.go:223] Registration of the systemd container factory successfully Nov 04 00:57:50.386992 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.386947 2579 factory.go:103] Registering Raw factory Nov 04 00:57:50.386992 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.386960 2579 manager.go:1199] Started watching for new ooms in manager Nov 04 00:57:50.387773 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.387760 2579 manager.go:319] Starting recovery of all containers Nov 04 00:57:50.388260 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.388239 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 04 00:57:50.399898 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.399876 2579 manager.go:324] Recovery completed Nov 04 00:57:50.403629 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.403615 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:50.406285 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.406270 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:50.406343 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.406298 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:50.406343 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.406308 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:50.406889 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.406875 2579 cpu_manager.go:226] "Starting" policy="none" Nov 04 00:57:50.406889 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.406889 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 00:57:50.406977 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.406908 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 00:57:50.406977 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.406882 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:50.407610 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.407595 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:50.407681 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.407624 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:50.407681 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.407637 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:50.409279 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.409266 2579 policy_none.go:50] "Start" Nov 04 00:57:50.409346 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.409284 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 00:57:50.409346 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.409301 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 00:57:50.411443 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.411430 2579 policy_none.go:44] "Start" Nov 04 00:57:50.415528 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.415432 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340c766c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-241.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.406284908 +0000 UTC m=+0.389822760,LastTimestamp:2026-11-04 00:57:50.406284908 +0000 UTC m=+0.389822760,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.427984 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.427928 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cbaa1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-241.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.406302369 +0000 UTC m=+0.389840220,LastTimestamp:2026-11-04 00:57:50.406302369 +0000 UTC m=+0.389840220,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.438715 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.438477 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cdec3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-241.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.406311619 +0000 UTC m=+0.389849472,LastTimestamp:2026-11-04 00:57:50.406311619 +0000 UTC m=+0.389849472,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.447280 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.447206 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340c766c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340c766c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-241.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.406284908 +0000 UTC m=+0.389822760,LastTimestamp:2026-11-04 00:57:50.407608458 +0000 UTC m=+0.391146309,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.452118 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.452093 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 00:57:50.454335 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.454321 2579 manager.go:341] "Starting Device Plugin manager" Nov 04 00:57:50.454423 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.454352 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 00:57:50.454423 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.454361 2579 server.go:87] "Starting device plugin registration server" Nov 04 00:57:50.454617 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.454605 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 00:57:50.454673 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.454619 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 00:57:50.454758 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.454726 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 00:57:50.454843 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.454808 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 00:57:50.454843 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.454819 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 00:57:50.455255 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.455236 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 00:57:50.455338 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.455283 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:50.458502 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.458441 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cbaa1\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cbaa1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-241.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.406302369 +0000 UTC m=+0.389840220,LastTimestamp:2026-11-04 00:57:50.407630279 +0000 UTC m=+0.391168132,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.471689 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.471635 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cdec3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cdec3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-241.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.406311619 +0000 UTC m=+0.389849472,LastTimestamp:2026-11-04 00:57:50.407641384 +0000 UTC m=+0.391179236,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.479745 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.479676 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a436e90c14 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated limits on system reserved cgroup /system.slice,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.454295572 +0000 UTC m=+0.437833410,LastTimestamp:2026-11-04 00:57:50.454295572 +0000 UTC m=+0.437833410,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.511454 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.495438 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a437138797 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.457079703 +0000 UTC m=+0.440617553,LastTimestamp:2026-11-04 00:57:50.457079703 +0000 UTC m=+0.440617553,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.555582 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.555532 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:50.556345 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.556332 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:50.556391 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.556362 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:50.556391 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.556379 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:50.556449 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.556408 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:50.565717 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.565654 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340c766c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340c766c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-241.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.406284908 +0000 UTC m=+0.389822760,LastTimestamp:2026-11-04 00:57:50.556345101 +0000 UTC m=+0.539882952,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.574022 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.574002 2579 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-140-241.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:50.574116 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.573980 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cbaa1\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cbaa1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-241.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.406302369 +0000 UTC m=+0.389840220,LastTimestamp:2026-11-04 00:57:50.556370234 +0000 UTC m=+0.539908094,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.580379 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.580347 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 00:57:50.581656 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.581639 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 00:57:50.581656 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.581658 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 00:57:50.581784 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.581692 2579 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 00:57:50.581784 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.581729 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 04 00:57:50.582808 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.582724 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cdec3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cdec3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-241.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.406311619 +0000 UTC m=+0.389849472,LastTimestamp:2026-11-04 00:57:50.5563847 +0000 UTC m=+0.539922554,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.595856 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.595836 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-241.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms" Nov 04 00:57:50.607164 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.607146 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 04 00:57:50.682354 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.682317 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal"] Nov 04 00:57:50.682418 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.682395 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:50.683203 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.683181 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:50.683282 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.683215 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:50.683282 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.683229 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:50.685379 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.685366 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:50.685529 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.685508 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:50.685585 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.685550 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:50.685675 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.685643 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f31320b40fb2fb40491617f2b2422d38-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal\" (UID: \"f31320b40fb2fb40491617f2b2422d38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:50.685710 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.685684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31320b40fb2fb40491617f2b2422d38-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal\" (UID: \"f31320b40fb2fb40491617f2b2422d38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:50.686117 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.686100 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:50.686209 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.686124 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:50.686209 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.686133 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:50.686209 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.686099 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:50.686209 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.686206 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:50.686329 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.686220 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:50.688249 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.688231 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:50.688327 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.688258 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:50.689022 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.688996 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:50.689117 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.689028 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:50.689117 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.689040 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:50.693547 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.693488 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340c766c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340c766c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-241.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.406284908 +0000 UTC m=+0.389822760,LastTimestamp:2026-11-04 00:57:50.68319968 +0000 UTC m=+0.666737536,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.702198 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.702141 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cbaa1\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cbaa1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-241.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.406302369 +0000 UTC m=+0.389840220,LastTimestamp:2026-11-04 00:57:50.683222504 +0000 UTC m=+0.666760360,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.715176 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.715159 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" node="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:50.717939 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.717920 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" node="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:50.722221 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.722163 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cdec3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cdec3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-241.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.406311619 +0000 UTC m=+0.389849472,LastTimestamp:2026-11-04 00:57:50.683233295 +0000 UTC m=+0.666771150,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.731066 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.731009 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340c766c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340c766c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-241.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.406284908 +0000 UTC m=+0.389822760,LastTimestamp:2026-11-04 00:57:50.686115354 +0000 UTC m=+0.669653205,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.739368 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.739312 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cbaa1\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cbaa1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-241.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.406302369 +0000 UTC m=+0.389840220,LastTimestamp:2026-11-04 00:57:50.686127998 +0000 UTC m=+0.669665849,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.749187 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.749125 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cdec3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cdec3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-241.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.406311619 +0000 UTC m=+0.389849472,LastTimestamp:2026-11-04 00:57:50.686137273 +0000 UTC m=+0.669675124,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.757885 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.757831 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340c766c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340c766c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-241.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.406284908 +0000 UTC m=+0.389822760,LastTimestamp:2026-11-04 00:57:50.686189462 +0000 UTC m=+0.669727318,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.766887 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.766826 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cbaa1\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cbaa1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-241.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.406302369 +0000 UTC m=+0.389840220,LastTimestamp:2026-11-04 00:57:50.686214276 +0000 UTC m=+0.669752128,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.774767 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.774750 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:50.775397 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.775380 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:50.775468 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.775410 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:50.775468 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.775424 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:50.775468 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.775444 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:50.777820 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.777740 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cdec3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cdec3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-241.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.406311619 +0000 UTC m=+0.389849472,LastTimestamp:2026-11-04 00:57:50.686225848 +0000 UTC m=+0.669763700,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.786145 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.786127 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf6706eb85e3c341c82e407ba659da4b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal\" (UID: \"cf6706eb85e3c341c82e407ba659da4b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:50.786249 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.786173 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f31320b40fb2fb40491617f2b2422d38-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal\" (UID: \"f31320b40fb2fb40491617f2b2422d38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:50.786249 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.786200 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31320b40fb2fb40491617f2b2422d38-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal\" (UID: \"f31320b40fb2fb40491617f2b2422d38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:50.786342 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.786255 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31320b40fb2fb40491617f2b2422d38-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal\" (UID: \"f31320b40fb2fb40491617f2b2422d38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:50.786342 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.786256 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f31320b40fb2fb40491617f2b2422d38-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal\" (UID: \"f31320b40fb2fb40491617f2b2422d38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:50.787266 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.787246 2579 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-140-241.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:50.787396 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.787338 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340c766c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340c766c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-241.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.406284908 +0000 UTC m=+0.389822760,LastTimestamp:2026-11-04 00:57:50.689012038 +0000 UTC m=+0.672549891,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.796508 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.796452 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cbaa1\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cbaa1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-241.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.406302369 +0000 UTC m=+0.389840220,LastTimestamp:2026-11-04 00:57:50.689033119 +0000 UTC m=+0.672570971,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.806362 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.806283 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cdec3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340cdec3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-241.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.406311619 +0000 UTC m=+0.389849472,LastTimestamp:2026-11-04 00:57:50.689045085 +0000 UTC m=+0.672582938,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.814812 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:50.814756 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340c766c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4340c766c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-241.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-241.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:50.406284908 +0000 UTC m=+0.389822760,LastTimestamp:2026-11-04 00:57:50.775396642 +0000 UTC m=+0.758934495,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:50.886318 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.886299 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf6706eb85e3c341c82e407ba659da4b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal\" (UID: \"cf6706eb85e3c341c82e407ba659da4b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:50.886382 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:50.886342 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf6706eb85e3c341c82e407ba659da4b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal\" (UID: \"cf6706eb85e3c341c82e407ba659da4b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:51.005020 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:51.004989 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-241.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="800ms" Nov 04 00:57:51.018138 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:51.018117 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:51.022801 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:51.022472 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:51.188285 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:51.188223 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:51.189099 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:51.189083 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:51.189174 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:51.189116 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:51.189174 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:51.189127 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:51.189174 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:51.189157 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:51.206230 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:51.206204 2579 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-140-241.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:51.359388 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:51.359359 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 04 00:57:51.394232 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:51.394208 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-241.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 04 00:57:51.499396 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:51.499347 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 04 00:57:51.551447 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:51.551425 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:51.552302 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:51.552285 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:51.552397 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:51.552314 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:51.552397 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:51.552324 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:51.680264 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:51.680239 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 04 00:57:51.732130 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:51.731551 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6706eb85e3c341c82e407ba659da4b.slice/crio-1b0a4b0b861d58a2c7583fedabd3e2e24771e738f0e5ece34c94211c52375518 WatchSource:0}: Error finding container 1b0a4b0b861d58a2c7583fedabd3e2e24771e738f0e5ece34c94211c52375518: Status 404 returned error can't find the container with id 1b0a4b0b861d58a2c7583fedabd3e2e24771e738f0e5ece34c94211c52375518 Nov 04 00:57:51.732314 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:57:51.732299 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31320b40fb2fb40491617f2b2422d38.slice/crio-652f3da9f495dbf0ada3f6694492cdcaf14feec9c4f9038817871fec701a498a WatchSource:0}: Error finding container 652f3da9f495dbf0ada3f6694492cdcaf14feec9c4f9038817871fec701a498a: Status 404 returned error can't find the container with id 652f3da9f495dbf0ada3f6694492cdcaf14feec9c4f9038817871fec701a498a Nov 04 00:57:51.737519 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:51.737504 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 00:57:51.747636 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:51.747544 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4836d8515 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal,UID:f31320b40fb2fb40491617f2b2422d38,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\",Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:51.738045717 +0000 UTC m=+1.721583568,LastTimestamp:2026-11-04 00:57:51.738045717 +0000 UTC m=+1.721583568,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:51.755674 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:51.755585 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4837fe67c kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal,UID:cf6706eb85e3c341c82e407ba659da4b,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e\",Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:51.7392503 +0000 UTC m=+1.722788138,LastTimestamp:2026-11-04 00:57:51.7392503 +0000 UTC m=+1.722788138,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:51.815036 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:51.815012 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-241.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="1.6s" Nov 04 00:57:51.960656 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:51.960629 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-241.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 04 00:57:52.006981 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:52.006938 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:52.008155 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:52.008137 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:52.008235 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:52.008166 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:52.008235 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:52.008176 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:52.008235 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:52.008205 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:52.026499 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:52.026478 2579 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-140-241.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:52.392315 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:52.392248 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-241.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 04 00:57:52.587104 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:52.587048 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" event={"ID":"f31320b40fb2fb40491617f2b2422d38","Type":"ContainerStarted","Data":"652f3da9f495dbf0ada3f6694492cdcaf14feec9c4f9038817871fec701a498a"} Nov 04 00:57:52.588235 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:52.588210 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal" event={"ID":"cf6706eb85e3c341c82e407ba659da4b","Type":"ContainerStarted","Data":"1b0a4b0b861d58a2c7583fedabd3e2e24771e738f0e5ece34c94211c52375518"} Nov 04 00:57:53.254582 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:53.254525 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 04 00:57:53.393333 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:53.393296 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-241.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 04 00:57:53.426757 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:53.426719 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-241.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="3.2s" Nov 04 00:57:53.494257 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:53.494220 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 04 00:57:53.545906 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:53.545791 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4ee7e673e openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal,UID:f31320b40fb2fb40491617f2b2422d38,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\" in 1.796s (1.796s including waiting). Image size: 272339155 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:53.534314302 +0000 UTC m=+3.517852161,LastTimestamp:2026-11-04 00:57:53.534314302 +0000 UTC m=+3.517852161,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:53.613749 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:53.613719 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:53.614965 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:53.614798 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:53.614965 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:53.614833 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:53.614965 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:53.614851 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:53.626623 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:53.626585 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:53.629214 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:53.628201 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:53.629214 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:53.628234 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:53.629214 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:53.628251 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:53.629214 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:53.628320 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:53.650118 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:53.650089 2579 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-140-241.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:53.710470 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:53.710363 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4f8552209 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal,UID:cf6706eb85e3c341c82e407ba659da4b,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e\" in 1.96s (1.96s including waiting). Image size: 294084393 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:53.699381769 +0000 UTC m=+3.682919632,LastTimestamp:2026-11-04 00:57:53.699381769 +0000 UTC m=+3.682919632,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:53.772227 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:53.772150 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4fc02d305 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal,UID:cf6706eb85e3c341c82e407ba659da4b,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Created,Message:Container created,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:53.761096453 +0000 UTC m=+3.744634307,LastTimestamp:2026-11-04 00:57:53.761096453 +0000 UTC m=+3.744634307,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:53.781849 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:53.781773 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4fc189609 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal,UID:f31320b40fb2fb40491617f2b2422d38,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Container created,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:53.762522633 +0000 UTC m=+3.746060487,LastTimestamp:2026-11-04 00:57:53.762522633 +0000 UTC m=+3.746060487,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:53.791690 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:53.791589 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4fc85b442 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal,UID:cf6706eb85e3c341c82e407ba659da4b,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Started,Message:Container started,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:53.769673794 +0000 UTC m=+3.753211639,LastTimestamp:2026-11-04 00:57:53.769673794 +0000 UTC m=+3.753211639,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:53.800771 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:53.800697 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal.18e4b1a4fca6ec6f openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal,UID:f31320b40fb2fb40491617f2b2422d38,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Container started,Source:EventSource{Component:kubelet,Host:ip-10-0-140-241.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:53.771850863 +0000 UTC m=+3.755388718,LastTimestamp:2026-11-04 00:57:53.771850863 +0000 UTC m=+3.755388718,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-241.us-east-2.compute.internal,}" Nov 04 00:57:54.213735 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:54.213657 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-241.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 04 00:57:54.392961 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:54.392939 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-241.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 04 00:57:54.594403 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:54.593977 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" event={"ID":"f31320b40fb2fb40491617f2b2422d38","Type":"ContainerStarted","Data":"4ae69cfd44ebd6c7ccc3e6476aaf952e8693a901e1e6226f89d0f0c3faa64b28"} Nov 04 00:57:54.594403 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:54.594271 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:54.595682 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:54.595660 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:54.595818 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:54.595696 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:54.595818 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:54.595708 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:54.595948 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:54.595910 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" node="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:54.596747 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:54.596720 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal" event={"ID":"cf6706eb85e3c341c82e407ba659da4b","Type":"ContainerStarted","Data":"012550e65193b9ba709455d17a7e4204dd8d2d1d452eefaa2eb517ed062ee1a4"} Nov 04 00:57:54.596841 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:54.596778 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:54.598246 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:54.598229 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:54.598297 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:54.598254 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:54.598297 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:54.598266 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:54.598418 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:54.598405 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" node="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:54.617493 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:54.617463 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pcck9" Nov 04 00:57:54.624903 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:54.624885 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:55.325109 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:55.325071 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 04 00:57:55.325266 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:55.325226 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 00:57:55.399605 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:55.399585 2579 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-140-241.us-east-2.compute.internal" not found Nov 04 00:57:55.414694 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:55.414680 2579 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-140-241.us-east-2.compute.internal" not found Nov 04 00:57:55.472019 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:55.472001 2579 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-140-241.us-east-2.compute.internal" not found Nov 04 00:57:55.598790 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:55.598726 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:55.599113 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:55.598726 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:55.599988 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:55.599965 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:55.600112 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:55.599996 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:55.600112 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:55.600009 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:55.600112 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:55.599974 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:55.600112 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:55.600088 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:55.600112 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:55.600103 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:55.600349 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:55.600331 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" node="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:55.600442 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:55.600426 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" node="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:55.621126 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:55.621086 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 00:52:55 +0000 UTC" deadline="2028-06-20 17:18:46.98718053 +0000 UTC" Nov 04 00:57:55.621126 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:55.621123 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14272h20m51.36606309s" Nov 04 00:57:55.729808 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:55.729788 2579 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-140-241.us-east-2.compute.internal" not found Nov 04 00:57:55.729808 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:55.729808 2579 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-140-241.us-east-2.compute.internal" not found Nov 04 00:57:55.767610 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:55.767589 2579 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-140-241.us-east-2.compute.internal" not found Nov 04 00:57:55.782234 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:55.782215 2579 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-140-241.us-east-2.compute.internal" not found Nov 04 00:57:55.839096 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:55.839070 2579 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-140-241.us-east-2.compute.internal" not found Nov 04 00:57:56.116432 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:56.116408 2579 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-140-241.us-east-2.compute.internal" not found Nov 04 00:57:56.116432 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:56.116431 2579 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-140-241.us-east-2.compute.internal" not found Nov 04 00:57:56.385739 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:56.385672 2579 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-140-241.us-east-2.compute.internal" not found Nov 04 00:57:56.399874 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:56.399853 2579 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-140-241.us-east-2.compute.internal" not found Nov 04 00:57:56.459905 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:56.459884 2579 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-140-241.us-east-2.compute.internal" not found Nov 04 00:57:56.634971 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:56.634943 2579 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-241.us-east-2.compute.internal\" not found" node="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:56.728300 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:56.728250 2579 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-140-241.us-east-2.compute.internal" not found Nov 04 00:57:56.728300 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:56.728269 2579 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-140-241.us-east-2.compute.internal" not found Nov 04 00:57:56.850787 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:56.850759 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:56.852028 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:56.852012 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:56.852098 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:56.852044 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:56.852098 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:56.852054 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:56.852098 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:56.852084 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:56.862691 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:56.862673 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:56.862778 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:56.862701 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-241.us-east-2.compute.internal\": node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:56.894124 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:56.894095 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:56.994229 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:56.994170 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:57.094823 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:57.094795 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:57.195594 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:57.195552 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:57.296143 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:57.296086 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:57.396260 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:57.396229 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:57.497454 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:57.497417 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:57.598388 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:57.598323 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:57.699018 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:57.698981 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:57.799547 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:57.799512 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:57.900199 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:57.900134 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:57.914346 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:57.914319 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:57.915478 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:57.915458 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:57.915586 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:57.915497 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:57.915586 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:57.915516 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:57.944880 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:57.944852 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:58.001215 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:58.001179 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:58.102002 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:58.101977 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:58.203073 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:58.203026 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:58.303713 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:58.303679 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:58.405103 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:58.405075 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:58.505912 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:58.505856 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:58.606100 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:58.606070 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:58.706228 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:58.706197 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:58.807652 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:58.807596 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:58.908922 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:58.908882 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:59.011000 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:59.010956 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:59.112111 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:59.112053 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:59.214280 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:59.214255 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:59.316794 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:59.316768 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:59.354457 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:59.354432 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:59.418510 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:59.418464 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:59.520358 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:57:59.520333 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:57:59.587874 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:59.587852 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:59.684805 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:59.684741 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:59.699500 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:59.699479 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:57:59.700368 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:59.700353 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:57:59.711998 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:57:59.711978 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:58:00.219797 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.219774 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:58:00.368783 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.368750 2579 apiserver.go:52] "Watching apiserver" Nov 04 00:58:00.377755 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.377733 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 00:58:00.378179 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.378156 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-dxbxk","openshift-multus/network-metrics-daemon-dsxl7","openshift-ovn-kubernetes/ovnkube-node-vst89","kube-system/konnectivity-agent-8d5qr","openshift-network-diagnostics/network-check-target-b4k7w","kube-system/global-pull-secret-syncer-5v9xx","openshift-dns/node-resolver-7vrj7","openshift-network-operator/iptables-alerter-jlf57","kube-system/kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n","openshift-cluster-node-tuning-operator/tuned-hlvm6","openshift-image-registry/node-ca-v8q47","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-mlx4q"] Nov 04 00:58:00.380960 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.380942 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7vrj7" Nov 04 00:58:00.383338 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.383314 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.384826 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.384758 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 00:58:00.384826 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.384761 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 00:58:00.384973 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.384843 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4k8lm\"" Nov 04 00:58:00.387250 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.387216 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:58:00.387432 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.387411 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ln4qv\"" Nov 04 00:58:00.387552 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.387535 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 00:58:00.387754 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.387736 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.389239 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.389029 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 04 00:58:00.389239 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.389170 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8d5qr" Nov 04 00:58:00.390677 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.390616 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7cbe8a51-0318-450b-adb2-4abfe1097883-hosts-file\") pod \"node-resolver-7vrj7\" (UID: \"7cbe8a51-0318-450b-adb2-4abfe1097883\") " pod="openshift-dns/node-resolver-7vrj7" Nov 04 00:58:00.390677 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.390671 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-sysctl-conf\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.390803 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.390694 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7s8t\" (UniqueName: \"kubernetes.io/projected/847ad3d1-d945-4281-9a5b-15420289c70d-kube-api-access-j7s8t\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.390803 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.390771 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-run-netns\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.390874 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.390825 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.390909 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.390891 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xttk5\" (UniqueName: \"kubernetes.io/projected/7cbe8a51-0318-450b-adb2-4abfe1097883-kube-api-access-xttk5\") pod \"node-resolver-7vrj7\" (UID: \"7cbe8a51-0318-450b-adb2-4abfe1097883\") " pod="openshift-dns/node-resolver-7vrj7" Nov 04 00:58:00.390953 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.390901 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 00:58:00.390953 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.390924 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-modprobe-d\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.391031 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.390977 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-systemd\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.391031 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391025 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-sys\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.391124 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391050 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-host\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.391124 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391085 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-run-ovn\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.391124 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391117 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-run-ovn-kubernetes\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.391284 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391139 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6eada63-a80f-4eff-8076-6f01b9efa23e-ovn-node-metrics-cert\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.391284 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391192 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-sysconfig\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.391284 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-sysctl-d\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.391284 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391267 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-lib-modules\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.391482 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391295 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9jwd5\"" Nov 04 00:58:00.391482 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391294 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/847ad3d1-d945-4281-9a5b-15420289c70d-tmp\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.391482 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391376 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-systemd-units\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.391482 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391414 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6eada63-a80f-4eff-8076-6f01b9efa23e-ovnkube-config\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.391482 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391455 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-run\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.391482 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391475 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-run-openvswitch\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.391807 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391495 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-node-log\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.391807 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391532 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6eada63-a80f-4eff-8076-6f01b9efa23e-env-overrides\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.391807 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391563 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 00:58:00.391807 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391579 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cbe8a51-0318-450b-adb2-4abfe1097883-tmp-dir\") pod \"node-resolver-7vrj7\" (UID: \"7cbe8a51-0318-450b-adb2-4abfe1097883\") " pod="openshift-dns/node-resolver-7vrj7" Nov 04 00:58:00.391807 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391627 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 00:58:00.391807 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391639 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-slash\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.391807 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391669 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 00:58:00.391807 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391689 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-var-lib-openvswitch\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.391807 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391711 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-cni-bin\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.391807 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391728 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-kubelet\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.391807 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391744 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-run-systemd\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.391807 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391778 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-cni-netd\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.391807 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391803 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-kubernetes\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.392331 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391819 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-var-lib-kubelet\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.392331 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/847ad3d1-d945-4281-9a5b-15420289c70d-etc-tuned\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.392331 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-etc-openvswitch\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.392331 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391919 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-log-socket\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.392331 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.391948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6eada63-a80f-4eff-8076-6f01b9efa23e-ovnkube-script-lib\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.392331 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.392005 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8dw2\" (UniqueName: \"kubernetes.io/projected/c6eada63-a80f-4eff-8076-6f01b9efa23e-kube-api-access-j8dw2\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.392331 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.392324 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:00.392705 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:00.392431 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4k7w" podUID="0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec" Nov 04 00:58:00.393098 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.393070 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 00:58:00.393216 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.393194 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 00:58:00.393281 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.393227 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 00:58:00.393338 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.393275 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 00:58:00.393338 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.393276 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-psr9r\"" Nov 04 00:58:00.394916 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.394894 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:00.395015 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:00.394973 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9xx" podUID="47fb874a-90bd-41f9-9a8c-0e920d789ca2" Nov 04 00:58:00.397393 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.397371 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.400177 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.400154 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jlf57" Nov 04 00:58:00.402810 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.402787 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:00.402901 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:00.402864 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsxl7" podUID="bb2c2fb8-f28c-430b-8210-db7032a7073b" Nov 04 00:58:00.405324 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.405303 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.407962 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.407937 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.410794 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.410772 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v8q47" Nov 04 00:58:00.428187 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.428167 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 00:58:00.428276 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.428220 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6xk65" Nov 04 00:58:00.436928 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.436901 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6xk65" Nov 04 00:58:00.484775 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.484723 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 00:58:00.492939 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.492913 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/47fb874a-90bd-41f9-9a8c-0e920d789ca2-dbus\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:00.493025 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.492941 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-host\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.493025 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493004 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-run-ovn-kubernetes\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.493025 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6eada63-a80f-4eff-8076-6f01b9efa23e-ovn-node-metrics-cert\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.493222 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493047 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-host\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.493222 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493050 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f784deee-bbd7-4e50-94d1-7bb64fc61b91-cni-binary-copy\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.493222 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493084 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-socket-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.493222 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493111 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-sysconfig\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.493222 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493127 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/847ad3d1-d945-4281-9a5b-15420289c70d-tmp\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.493222 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493137 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-run-ovn-kubernetes\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.493222 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-systemd-units\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.493222 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493192 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.493222 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493217 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-sysconfig\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.493763 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493221 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/402ccc5f-d755-4061-a96b-8ddc0495b711-konnectivity-ca\") pod \"konnectivity-agent-8d5qr\" (UID: \"402ccc5f-d755-4061-a96b-8ddc0495b711\") " pod="kube-system/konnectivity-agent-8d5qr" Nov 04 00:58:00.493763 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493259 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-systemd-units\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.493763 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493264 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f784deee-bbd7-4e50-94d1-7bb64fc61b91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.493763 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493322 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/565a71a6-1ce7-4078-90fc-f2ccadaa0d17-serviceca\") pod \"node-ca-v8q47\" (UID: \"565a71a6-1ce7-4078-90fc-f2ccadaa0d17\") " pod="openshift-image-registry/node-ca-v8q47" Nov 04 00:58:00.493763 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493363 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-sys-fs\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.493763 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-cni-bin\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.493763 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493399 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 00:58:00.493763 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493423 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-cni-bin\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.493763 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493438 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htztj\" (UniqueName: \"kubernetes.io/projected/f784deee-bbd7-4e50-94d1-7bb64fc61b91-kube-api-access-htztj\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.493763 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493495 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-system-cni-dir\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.493763 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493525 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npmf5\" (UniqueName: \"kubernetes.io/projected/565a71a6-1ce7-4078-90fc-f2ccadaa0d17-kube-api-access-npmf5\") pod \"node-ca-v8q47\" (UID: \"565a71a6-1ce7-4078-90fc-f2ccadaa0d17\") " pod="openshift-image-registry/node-ca-v8q47" Nov 04 00:58:00.493763 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493590 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6083261-f4cb-41ee-999d-8c5406473ff8-iptables-alerter-script\") pod \"iptables-alerter-jlf57\" (UID: \"f6083261-f4cb-41ee-999d-8c5406473ff8\") " pod="openshift-network-operator/iptables-alerter-jlf57" Nov 04 00:58:00.493763 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493617 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-etc-selinux\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.493763 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-kubelet\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.493763 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493670 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-socket-dir-parent\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.493763 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493690 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-kubelet\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.493763 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493713 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:00.494753 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-kubernetes\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.494753 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493792 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac5e9e6b-7405-4c03-a149-4bf353209f24-cni-binary-copy\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.494753 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493841 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-kubernetes\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.494753 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493876 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-run-multus-certs\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.494753 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493913 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/565a71a6-1ce7-4078-90fc-f2ccadaa0d17-host\") pod \"node-ca-v8q47\" (UID: \"565a71a6-1ce7-4078-90fc-f2ccadaa0d17\") " pod="openshift-image-registry/node-ca-v8q47" Nov 04 00:58:00.494753 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.493978 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdklr\" (UniqueName: \"kubernetes.io/projected/bb2c2fb8-f28c-430b-8210-db7032a7073b-kube-api-access-pdklr\") pod \"network-metrics-daemon-dsxl7\" (UID: \"bb2c2fb8-f28c-430b-8210-db7032a7073b\") " pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:00.494753 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494008 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-run-netns\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.494753 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494025 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.494753 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494069 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-etc-kubernetes\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.494753 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494102 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8ms4\" (UniqueName: \"kubernetes.io/projected/f6083261-f4cb-41ee-999d-8c5406473ff8-kube-api-access-z8ms4\") pod \"iptables-alerter-jlf57\" (UID: \"f6083261-f4cb-41ee-999d-8c5406473ff8\") " pod="openshift-network-operator/iptables-alerter-jlf57" Nov 04 00:58:00.494753 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494119 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-run-netns\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.494753 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-device-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.494753 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494159 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xttk5\" (UniqueName: \"kubernetes.io/projected/7cbe8a51-0318-450b-adb2-4abfe1097883-kube-api-access-xttk5\") pod \"node-resolver-7vrj7\" (UID: \"7cbe8a51-0318-450b-adb2-4abfe1097883\") " pod="openshift-dns/node-resolver-7vrj7" Nov 04 00:58:00.494753 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-modprobe-d\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.494753 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494242 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.494753 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494272 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-systemd\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.494753 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-sys\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.495810 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494325 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-modprobe-d\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.495810 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494335 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-systemd\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.495810 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-run-ovn\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.495810 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494394 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f784deee-bbd7-4e50-94d1-7bb64fc61b91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.495810 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494439 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-sys\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.495810 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494438 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-sysctl-d\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.495810 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494463 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-run-ovn\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.495810 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494502 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-lib-modules\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.495810 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6eada63-a80f-4eff-8076-6f01b9efa23e-ovnkube-config\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.495810 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/402ccc5f-d755-4061-a96b-8ddc0495b711-agent-certs\") pod \"konnectivity-agent-8d5qr\" (UID: \"402ccc5f-d755-4061-a96b-8ddc0495b711\") " pod="kube-system/konnectivity-agent-8d5qr" Nov 04 00:58:00.495810 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494599 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-sysctl-d\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.495810 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494609 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-cni-dir\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.495810 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494603 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 00:58:00.495810 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494634 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-run\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.495810 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494671 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-run-openvswitch\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.495810 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494724 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-lib-modules\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.495810 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494753 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-node-log\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.495810 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494765 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 00:58:00.495810 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494779 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-run-openvswitch\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.496771 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494781 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6eada63-a80f-4eff-8076-6f01b9efa23e-env-overrides\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.496771 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494814 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-node-log\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.496771 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494820 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-conf-dir\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.496771 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494848 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs\") pod \"network-metrics-daemon-dsxl7\" (UID: \"bb2c2fb8-f28c-430b-8210-db7032a7073b\") " pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:00.496771 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-registration-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.496771 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494902 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 00:58:00.496771 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494902 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-run\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.496771 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494936 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f784deee-bbd7-4e50-94d1-7bb64fc61b91-system-cni-dir\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.496771 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494981 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f784deee-bbd7-4e50-94d1-7bb64fc61b91-cnibin\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.496771 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.494989 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 00:58:00.496771 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f784deee-bbd7-4e50-94d1-7bb64fc61b91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.496771 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-hostroot\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.496771 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495054 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdntp\" (UniqueName: \"kubernetes.io/projected/975903c9-9da2-4971-a50b-7bcc37df6b34-kube-api-access-bdntp\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.496771 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495079 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cbe8a51-0318-450b-adb2-4abfe1097883-tmp-dir\") pod \"node-resolver-7vrj7\" (UID: \"7cbe8a51-0318-450b-adb2-4abfe1097883\") " pod="openshift-dns/node-resolver-7vrj7" Nov 04 00:58:00.496771 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-slash\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.496771 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495159 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-var-lib-openvswitch\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.496771 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495167 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 00:58:00.496771 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495182 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-var-lib-cni-multus\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.497618 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/47fb874a-90bd-41f9-9a8c-0e920d789ca2-kubelet-config\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:00.497618 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495206 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-slash\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.497618 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-run-systemd\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.497618 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495224 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-var-lib-openvswitch\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.497618 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495161 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 00:58:00.497618 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-cni-netd\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.497618 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495286 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-cnibin\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.497618 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495314 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-var-lib-cni-bin\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.497618 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495322 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-cni-netd\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.497618 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495288 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-run-systemd\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.497618 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495348 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-daemon-config\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.497618 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495391 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzwsb\" (UniqueName: \"kubernetes.io/projected/ac5e9e6b-7405-4c03-a149-4bf353209f24-kube-api-access-xzwsb\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.497618 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495421 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llvc6\" (UniqueName: \"kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6\") pod \"network-check-target-b4k7w\" (UID: \"0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec\") " pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:00.497618 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495434 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cbe8a51-0318-450b-adb2-4abfe1097883-tmp-dir\") pod \"node-resolver-7vrj7\" (UID: \"7cbe8a51-0318-450b-adb2-4abfe1097883\") " pod="openshift-dns/node-resolver-7vrj7" Nov 04 00:58:00.497618 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495454 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6083261-f4cb-41ee-999d-8c5406473ff8-host-slash\") pod \"iptables-alerter-jlf57\" (UID: \"f6083261-f4cb-41ee-999d-8c5406473ff8\") " pod="openshift-network-operator/iptables-alerter-jlf57" Nov 04 00:58:00.497618 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495483 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-var-lib-kubelet\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.497618 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495507 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:58:00.497618 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495533 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/847ad3d1-d945-4281-9a5b-15420289c70d-etc-tuned\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.497618 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495556 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 00:58:00.498216 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495561 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-etc-openvswitch\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.498216 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495603 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 00:58:00.498216 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495653 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-log-socket\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.498216 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495690 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6eada63-a80f-4eff-8076-6f01b9efa23e-ovnkube-script-lib\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.498216 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495716 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8dw2\" (UniqueName: \"kubernetes.io/projected/c6eada63-a80f-4eff-8076-6f01b9efa23e-kube-api-access-j8dw2\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.498216 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495716 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-etc-openvswitch\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.498216 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495772 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-run-netns\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.498216 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495795 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 00:58:00.498216 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495803 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-var-lib-kubelet\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.498216 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7cbe8a51-0318-450b-adb2-4abfe1097883-hosts-file\") pod \"node-resolver-7vrj7\" (UID: \"7cbe8a51-0318-450b-adb2-4abfe1097883\") " pod="openshift-dns/node-resolver-7vrj7" Nov 04 00:58:00.498216 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-sysctl-conf\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.498216 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495880 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7s8t\" (UniqueName: \"kubernetes.io/projected/847ad3d1-d945-4281-9a5b-15420289c70d-kube-api-access-j7s8t\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.498216 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495882 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 00:58:00.498216 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495893 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6eada63-a80f-4eff-8076-6f01b9efa23e-env-overrides\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.498216 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495959 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-var-lib-kubelet\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.498216 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495966 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-run-k8s-cni-cncf-io\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.498216 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.495990 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6eada63-a80f-4eff-8076-6f01b9efa23e-ovnkube-config\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.498216 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.496016 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-sysctl-conf\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.498216 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.496066 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7cbe8a51-0318-450b-adb2-4abfe1097883-hosts-file\") pod \"node-resolver-7vrj7\" (UID: \"7cbe8a51-0318-450b-adb2-4abfe1097883\") " pod="openshift-dns/node-resolver-7vrj7" Nov 04 00:58:00.498824 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.496086 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-log-socket\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.498824 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.496370 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 00:58:00.498824 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.496662 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6eada63-a80f-4eff-8076-6f01b9efa23e-ovnkube-script-lib\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.498824 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.497048 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zgrbh\"" Nov 04 00:58:00.498824 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.497094 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-xdt54\"" Nov 04 00:58:00.498824 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.497105 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 00:58:00.498824 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.497145 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zx855\"" Nov 04 00:58:00.498824 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.497163 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sknxk\"" Nov 04 00:58:00.498824 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.497210 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kqwbk\"" Nov 04 00:58:00.498824 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.497326 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 00:58:00.498824 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.497396 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 00:58:00.498824 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.497438 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/847ad3d1-d945-4281-9a5b-15420289c70d-tmp\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.498824 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.497785 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6eada63-a80f-4eff-8076-6f01b9efa23e-ovn-node-metrics-cert\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.498824 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.498257 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/847ad3d1-d945-4281-9a5b-15420289c70d-etc-tuned\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.503798 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.503774 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xttk5\" (UniqueName: \"kubernetes.io/projected/7cbe8a51-0318-450b-adb2-4abfe1097883-kube-api-access-xttk5\") pod \"node-resolver-7vrj7\" (UID: \"7cbe8a51-0318-450b-adb2-4abfe1097883\") " pod="openshift-dns/node-resolver-7vrj7" Nov 04 00:58:00.504869 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.504847 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8dw2\" (UniqueName: \"kubernetes.io/projected/c6eada63-a80f-4eff-8076-6f01b9efa23e-kube-api-access-j8dw2\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.505073 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.505056 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7s8t\" (UniqueName: \"kubernetes.io/projected/847ad3d1-d945-4281-9a5b-15420289c70d-kube-api-access-j7s8t\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.596244 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/402ccc5f-d755-4061-a96b-8ddc0495b711-agent-certs\") pod \"konnectivity-agent-8d5qr\" (UID: \"402ccc5f-d755-4061-a96b-8ddc0495b711\") " pod="kube-system/konnectivity-agent-8d5qr" Nov 04 00:58:00.596354 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596246 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-cni-dir\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.596354 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596266 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-conf-dir\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.596354 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs\") pod \"network-metrics-daemon-dsxl7\" (UID: \"bb2c2fb8-f28c-430b-8210-db7032a7073b\") " pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:00.596354 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596296 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-registration-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.596354 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596320 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f784deee-bbd7-4e50-94d1-7bb64fc61b91-system-cni-dir\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.596354 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596343 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f784deee-bbd7-4e50-94d1-7bb64fc61b91-cnibin\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.596675 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f784deee-bbd7-4e50-94d1-7bb64fc61b91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.596675 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-hostroot\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.596675 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:00.596407 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:58:00.596675 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596418 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-conf-dir\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.596675 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596424 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-cni-dir\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.596675 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596420 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdntp\" (UniqueName: \"kubernetes.io/projected/975903c9-9da2-4971-a50b-7bcc37df6b34-kube-api-access-bdntp\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.596675 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596417 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f784deee-bbd7-4e50-94d1-7bb64fc61b91-system-cni-dir\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.596675 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:00.596472 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs podName:bb2c2fb8-f28c-430b-8210-db7032a7073b nodeName:}" failed. No retries permitted until 2026-11-04 00:58:01.096451565 +0000 UTC m=+11.079989427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs") pod "network-metrics-daemon-dsxl7" (UID: "bb2c2fb8-f28c-430b-8210-db7032a7073b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:58:00.596675 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596491 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-hostroot\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.596675 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596515 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-registration-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.596675 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596569 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f784deee-bbd7-4e50-94d1-7bb64fc61b91-cnibin\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.596675 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-var-lib-cni-multus\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.597294 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/47fb874a-90bd-41f9-9a8c-0e920d789ca2-kubelet-config\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:00.597294 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-cnibin\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.597294 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596736 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-var-lib-cni-bin\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.597294 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596738 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-var-lib-cni-multus\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.597294 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596761 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-daemon-config\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.597294 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596787 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzwsb\" (UniqueName: \"kubernetes.io/projected/ac5e9e6b-7405-4c03-a149-4bf353209f24-kube-api-access-xzwsb\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.597294 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596790 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-cnibin\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.597294 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596758 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/47fb874a-90bd-41f9-9a8c-0e920d789ca2-kubelet-config\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:00.597294 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596790 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-var-lib-cni-bin\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.597294 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596812 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llvc6\" (UniqueName: \"kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6\") pod \"network-check-target-b4k7w\" (UID: \"0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec\") " pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:00.597294 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.596964 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6083261-f4cb-41ee-999d-8c5406473ff8-host-slash\") pod \"iptables-alerter-jlf57\" (UID: \"f6083261-f4cb-41ee-999d-8c5406473ff8\") " pod="openshift-network-operator/iptables-alerter-jlf57" Nov 04 00:58:00.597294 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-run-netns\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.597294 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597078 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6083261-f4cb-41ee-999d-8c5406473ff8-host-slash\") pod \"iptables-alerter-jlf57\" (UID: \"f6083261-f4cb-41ee-999d-8c5406473ff8\") " pod="openshift-network-operator/iptables-alerter-jlf57" Nov 04 00:58:00.597294 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597082 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-var-lib-kubelet\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.597294 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597091 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f784deee-bbd7-4e50-94d1-7bb64fc61b91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.597294 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597122 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-run-netns\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.597294 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597139 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-var-lib-kubelet\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.597294 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597231 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-run-k8s-cni-cncf-io\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.598059 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-run-k8s-cni-cncf-io\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.598059 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597319 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-daemon-config\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.598059 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597337 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/47fb874a-90bd-41f9-9a8c-0e920d789ca2-dbus\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:00.598059 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f784deee-bbd7-4e50-94d1-7bb64fc61b91-cni-binary-copy\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.598059 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-socket-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.598059 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597422 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.598059 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597457 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/402ccc5f-d755-4061-a96b-8ddc0495b711-konnectivity-ca\") pod \"konnectivity-agent-8d5qr\" (UID: \"402ccc5f-d755-4061-a96b-8ddc0495b711\") " pod="kube-system/konnectivity-agent-8d5qr" Nov 04 00:58:00.598059 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597472 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/47fb874a-90bd-41f9-9a8c-0e920d789ca2-dbus\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:00.598059 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597484 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f784deee-bbd7-4e50-94d1-7bb64fc61b91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.598059 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597506 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-socket-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.598059 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597514 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/565a71a6-1ce7-4078-90fc-f2ccadaa0d17-serviceca\") pod \"node-ca-v8q47\" (UID: \"565a71a6-1ce7-4078-90fc-f2ccadaa0d17\") " pod="openshift-image-registry/node-ca-v8q47" Nov 04 00:58:00.598059 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597512 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.598059 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597549 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-sys-fs\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.598059 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597601 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htztj\" (UniqueName: \"kubernetes.io/projected/f784deee-bbd7-4e50-94d1-7bb64fc61b91-kube-api-access-htztj\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.598059 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-system-cni-dir\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.598059 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597648 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-sys-fs\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.598059 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npmf5\" (UniqueName: \"kubernetes.io/projected/565a71a6-1ce7-4078-90fc-f2ccadaa0d17-kube-api-access-npmf5\") pod \"node-ca-v8q47\" (UID: \"565a71a6-1ce7-4078-90fc-f2ccadaa0d17\") " pod="openshift-image-registry/node-ca-v8q47" Nov 04 00:58:00.598684 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597703 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6083261-f4cb-41ee-999d-8c5406473ff8-iptables-alerter-script\") pod \"iptables-alerter-jlf57\" (UID: \"f6083261-f4cb-41ee-999d-8c5406473ff8\") " pod="openshift-network-operator/iptables-alerter-jlf57" Nov 04 00:58:00.598684 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597711 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f784deee-bbd7-4e50-94d1-7bb64fc61b91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.598684 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597741 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-etc-selinux\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.598684 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597768 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-socket-dir-parent\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.598684 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:00.598684 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:00.597887 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:58:00.598684 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:00.597932 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret podName:47fb874a-90bd-41f9-9a8c-0e920d789ca2 nodeName:}" failed. No retries permitted until 2026-11-04 00:58:01.09791633 +0000 UTC m=+11.081454176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret") pod "global-pull-secret-syncer-5v9xx" (UID: "47fb874a-90bd-41f9-9a8c-0e920d789ca2") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:58:00.598684 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.597999 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-etc-selinux\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.598684 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.598015 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-system-cni-dir\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.598684 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.598039 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac5e9e6b-7405-4c03-a149-4bf353209f24-cni-binary-copy\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.598684 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.598043 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-socket-dir-parent\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.598684 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.598057 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-run-multus-certs\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.598684 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.598074 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/565a71a6-1ce7-4078-90fc-f2ccadaa0d17-host\") pod \"node-ca-v8q47\" (UID: \"565a71a6-1ce7-4078-90fc-f2ccadaa0d17\") " pod="openshift-image-registry/node-ca-v8q47" Nov 04 00:58:00.598684 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.598093 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdklr\" (UniqueName: \"kubernetes.io/projected/bb2c2fb8-f28c-430b-8210-db7032a7073b-kube-api-access-pdklr\") pod \"network-metrics-daemon-dsxl7\" (UID: \"bb2c2fb8-f28c-430b-8210-db7032a7073b\") " pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:00.598684 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.598112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-etc-kubernetes\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.598684 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.598157 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/565a71a6-1ce7-4078-90fc-f2ccadaa0d17-host\") pod \"node-ca-v8q47\" (UID: \"565a71a6-1ce7-4078-90fc-f2ccadaa0d17\") " pod="openshift-image-registry/node-ca-v8q47" Nov 04 00:58:00.598684 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.598158 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-run-multus-certs\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.599284 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.598170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8ms4\" (UniqueName: \"kubernetes.io/projected/f6083261-f4cb-41ee-999d-8c5406473ff8-kube-api-access-z8ms4\") pod \"iptables-alerter-jlf57\" (UID: \"f6083261-f4cb-41ee-999d-8c5406473ff8\") " pod="openshift-network-operator/iptables-alerter-jlf57" Nov 04 00:58:00.599284 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.598202 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-device-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.599284 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.598224 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f784deee-bbd7-4e50-94d1-7bb64fc61b91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.599284 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.598226 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-etc-kubernetes\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.599284 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.598307 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-device-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.599284 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.598620 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f784deee-bbd7-4e50-94d1-7bb64fc61b91-cni-binary-copy\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.599284 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.598855 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/402ccc5f-d755-4061-a96b-8ddc0495b711-konnectivity-ca\") pod \"konnectivity-agent-8d5qr\" (UID: \"402ccc5f-d755-4061-a96b-8ddc0495b711\") " pod="kube-system/konnectivity-agent-8d5qr" Nov 04 00:58:00.599284 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.598889 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6083261-f4cb-41ee-999d-8c5406473ff8-iptables-alerter-script\") pod \"iptables-alerter-jlf57\" (UID: \"f6083261-f4cb-41ee-999d-8c5406473ff8\") " pod="openshift-network-operator/iptables-alerter-jlf57" Nov 04 00:58:00.599284 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.598900 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/565a71a6-1ce7-4078-90fc-f2ccadaa0d17-serviceca\") pod \"node-ca-v8q47\" (UID: \"565a71a6-1ce7-4078-90fc-f2ccadaa0d17\") " pod="openshift-image-registry/node-ca-v8q47" Nov 04 00:58:00.599284 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.598921 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f784deee-bbd7-4e50-94d1-7bb64fc61b91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.599284 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.598971 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac5e9e6b-7405-4c03-a149-4bf353209f24-cni-binary-copy\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.599751 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.599732 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/402ccc5f-d755-4061-a96b-8ddc0495b711-agent-certs\") pod \"konnectivity-agent-8d5qr\" (UID: \"402ccc5f-d755-4061-a96b-8ddc0495b711\") " pod="kube-system/konnectivity-agent-8d5qr" Nov 04 00:58:00.606994 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:00.606974 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:58:00.606994 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:00.606998 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:58:00.607166 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:00.607012 2579 projected.go:196] Error preparing data for projected volume kube-api-access-llvc6 for pod openshift-network-diagnostics/network-check-target-b4k7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:58:00.607166 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:00.607070 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6 podName:0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec nodeName:}" failed. No retries permitted until 2026-11-04 00:58:01.107055324 +0000 UTC m=+11.090593164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-llvc6" (UniqueName: "kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6") pod "network-check-target-b4k7w" (UID: "0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:58:00.608478 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.608454 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzwsb\" (UniqueName: \"kubernetes.io/projected/ac5e9e6b-7405-4c03-a149-4bf353209f24-kube-api-access-xzwsb\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.608588 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.608545 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdntp\" (UniqueName: \"kubernetes.io/projected/975903c9-9da2-4971-a50b-7bcc37df6b34-kube-api-access-bdntp\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.609189 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.609164 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npmf5\" (UniqueName: \"kubernetes.io/projected/565a71a6-1ce7-4078-90fc-f2ccadaa0d17-kube-api-access-npmf5\") pod \"node-ca-v8q47\" (UID: \"565a71a6-1ce7-4078-90fc-f2ccadaa0d17\") " pod="openshift-image-registry/node-ca-v8q47" Nov 04 00:58:00.609310 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.609262 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8ms4\" (UniqueName: \"kubernetes.io/projected/f6083261-f4cb-41ee-999d-8c5406473ff8-kube-api-access-z8ms4\") pod \"iptables-alerter-jlf57\" (UID: \"f6083261-f4cb-41ee-999d-8c5406473ff8\") " pod="openshift-network-operator/iptables-alerter-jlf57" Nov 04 00:58:00.609532 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.609515 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdklr\" (UniqueName: \"kubernetes.io/projected/bb2c2fb8-f28c-430b-8210-db7032a7073b-kube-api-access-pdklr\") pod \"network-metrics-daemon-dsxl7\" (UID: \"bb2c2fb8-f28c-430b-8210-db7032a7073b\") " pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:00.611030 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.611014 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htztj\" (UniqueName: \"kubernetes.io/projected/f784deee-bbd7-4e50-94d1-7bb64fc61b91-kube-api-access-htztj\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.611935 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.611895 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal" podStartSLOduration=1.611885291 podStartE2EDuration="1.611885291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:00.596448142 +0000 UTC m=+10.579986007" watchObservedRunningTime="2026-11-04 00:58:00.611885291 +0000 UTC m=+10.595423156" Nov 04 00:58:00.700138 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.700110 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7vrj7" Nov 04 00:58:00.706921 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.706903 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:00.715223 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:58:00.715198 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod847ad3d1_d945_4281_9a5b_15420289c70d.slice/crio-1ff8d5a10c88fcfb0fa26a96783e8067f5a9915e76562fcbd7995fd130f4b2de WatchSource:0}: Error finding container 1ff8d5a10c88fcfb0fa26a96783e8067f5a9915e76562fcbd7995fd130f4b2de: Status 404 returned error can't find the container with id 1ff8d5a10c88fcfb0fa26a96783e8067f5a9915e76562fcbd7995fd130f4b2de Nov 04 00:58:00.727825 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.727806 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8d5qr" Nov 04 00:58:00.734910 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.734861 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:00.736678 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:58:00.736656 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402ccc5f_d755_4061_a96b_8ddc0495b711.slice/crio-95ffdb116aae4604d44141563ca1e3486d047bf765addbbb136234608ae2b4c4 WatchSource:0}: Error finding container 95ffdb116aae4604d44141563ca1e3486d047bf765addbbb136234608ae2b4c4: Status 404 returned error can't find the container with id 95ffdb116aae4604d44141563ca1e3486d047bf765addbbb136234608ae2b4c4 Nov 04 00:58:00.736678 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.736662 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jlf57" Nov 04 00:58:00.743450 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:58:00.743414 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6eada63_a80f_4eff_8076_6f01b9efa23e.slice/crio-2c2ff0ae8b6107ced8dc11426662d452592f20733f317e1f9d573ff3f9f48ddb WatchSource:0}: Error finding container 2c2ff0ae8b6107ced8dc11426662d452592f20733f317e1f9d573ff3f9f48ddb: Status 404 returned error can't find the container with id 2c2ff0ae8b6107ced8dc11426662d452592f20733f317e1f9d573ff3f9f48ddb Nov 04 00:58:00.746426 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:58:00.746351 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6083261_f4cb_41ee_999d_8c5406473ff8.slice/crio-9b097049413e19ac6bbccd2cd05485546e0b37baf228f313ac7f97a283096122 WatchSource:0}: Error finding container 9b097049413e19ac6bbccd2cd05485546e0b37baf228f313ac7f97a283096122: Status 404 returned error can't find the container with id 9b097049413e19ac6bbccd2cd05485546e0b37baf228f313ac7f97a283096122 Nov 04 00:58:00.750608 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.750591 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dxbxk" Nov 04 00:58:00.752802 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.752783 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:00.754378 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.754362 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v8q47" Nov 04 00:58:00.761023 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:58:00.761001 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5e9e6b_7405_4c03_a149_4bf353209f24.slice/crio-3342c8d9052dfe9f3803d0db938c962dc58657ee79a9802b49e5be497a29772b WatchSource:0}: Error finding container 3342c8d9052dfe9f3803d0db938c962dc58657ee79a9802b49e5be497a29772b: Status 404 returned error can't find the container with id 3342c8d9052dfe9f3803d0db938c962dc58657ee79a9802b49e5be497a29772b Nov 04 00:58:00.761587 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:00.761578 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:00.764679 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:58:00.764646 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975903c9_9da2_4971_a50b_7bcc37df6b34.slice/crio-ac7c69deda762ae8537940d4a40a5047556167936c3d3932d3fc617ce507b23d WatchSource:0}: Error finding container ac7c69deda762ae8537940d4a40a5047556167936c3d3932d3fc617ce507b23d: Status 404 returned error can't find the container with id ac7c69deda762ae8537940d4a40a5047556167936c3d3932d3fc617ce507b23d Nov 04 00:58:00.767012 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:58:00.766995 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565a71a6_1ce7_4078_90fc_f2ccadaa0d17.slice/crio-116018e6f7a57844a290bbeafac250cf3eec2bd16bd247250c1589e0ee539b7c WatchSource:0}: Error finding container 116018e6f7a57844a290bbeafac250cf3eec2bd16bd247250c1589e0ee539b7c: Status 404 returned error can't find the container with id 116018e6f7a57844a290bbeafac250cf3eec2bd16bd247250c1589e0ee539b7c Nov 04 00:58:00.773354 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:58:00.773332 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf784deee_bbd7_4e50_94d1_7bb64fc61b91.slice/crio-4832f0e32b037011ba261e4a227f95b569e553145a1aec19d3bd81bebceef542 WatchSource:0}: Error finding container 4832f0e32b037011ba261e4a227f95b569e553145a1aec19d3bd81bebceef542: Status 404 returned error can't find the container with id 4832f0e32b037011ba261e4a227f95b569e553145a1aec19d3bd81bebceef542 Nov 04 00:58:01.102821 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:01.102762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs\") pod \"network-metrics-daemon-dsxl7\" (UID: \"bb2c2fb8-f28c-430b-8210-db7032a7073b\") " pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:01.102957 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:01.102821 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:01.102957 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:01.102914 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:58:01.102957 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:01.102930 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:58:01.102957 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:01.102959 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret podName:47fb874a-90bd-41f9-9a8c-0e920d789ca2 nodeName:}" failed. No retries permitted until 2026-11-04 00:58:02.102946256 +0000 UTC m=+12.086484111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret") pod "global-pull-secret-syncer-5v9xx" (UID: "47fb874a-90bd-41f9-9a8c-0e920d789ca2") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:58:01.103121 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:01.102987 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs podName:bb2c2fb8-f28c-430b-8210-db7032a7073b nodeName:}" failed. No retries permitted until 2026-11-04 00:58:02.102969944 +0000 UTC m=+12.086507795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs") pod "network-metrics-daemon-dsxl7" (UID: "bb2c2fb8-f28c-430b-8210-db7032a7073b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:58:01.204042 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:01.204017 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llvc6\" (UniqueName: \"kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6\") pod \"network-check-target-b4k7w\" (UID: \"0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec\") " pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:01.204154 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:01.204142 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:58:01.204198 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:01.204158 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:58:01.204198 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:01.204167 2579 projected.go:196] Error preparing data for projected volume kube-api-access-llvc6 for pod openshift-network-diagnostics/network-check-target-b4k7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:58:01.204265 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:01.204205 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6 podName:0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec nodeName:}" failed. No retries permitted until 2026-11-04 00:58:02.204194704 +0000 UTC m=+12.187732563 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-llvc6" (UniqueName: "kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6") pod "network-check-target-b4k7w" (UID: "0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:58:01.440814 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:01.440709 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 00:53:00 +0000 UTC" deadline="2028-08-05 07:25:37.211806472 +0000 UTC" Nov 04 00:58:01.440814 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:01.440766 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15366h27m35.77104402s" Nov 04 00:58:01.583350 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:01.582845 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:01.583350 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:01.582994 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsxl7" podUID="bb2c2fb8-f28c-430b-8210-db7032a7073b" Nov 04 00:58:01.617043 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:01.616344 2579 generic.go:356] "Generic (PLEG): container finished" podID="f31320b40fb2fb40491617f2b2422d38" containerID="4ae69cfd44ebd6c7ccc3e6476aaf952e8693a901e1e6226f89d0f0c3faa64b28" exitCode=0 Nov 04 00:58:01.617043 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:01.616475 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" event={"ID":"f31320b40fb2fb40491617f2b2422d38","Type":"ContainerDied","Data":"4ae69cfd44ebd6c7ccc3e6476aaf952e8693a901e1e6226f89d0f0c3faa64b28"} Nov 04 00:58:01.636867 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:01.636802 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v8q47" event={"ID":"565a71a6-1ce7-4078-90fc-f2ccadaa0d17","Type":"ContainerStarted","Data":"116018e6f7a57844a290bbeafac250cf3eec2bd16bd247250c1589e0ee539b7c"} Nov 04 00:58:01.650665 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:01.650624 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" event={"ID":"975903c9-9da2-4971-a50b-7bcc37df6b34","Type":"ContainerStarted","Data":"ac7c69deda762ae8537940d4a40a5047556167936c3d3932d3fc617ce507b23d"} Nov 04 00:58:01.660336 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:01.660308 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dxbxk" event={"ID":"ac5e9e6b-7405-4c03-a149-4bf353209f24","Type":"ContainerStarted","Data":"3342c8d9052dfe9f3803d0db938c962dc58657ee79a9802b49e5be497a29772b"} Nov 04 00:58:01.678310 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:01.678277 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jlf57" event={"ID":"f6083261-f4cb-41ee-999d-8c5406473ff8","Type":"ContainerStarted","Data":"9b097049413e19ac6bbccd2cd05485546e0b37baf228f313ac7f97a283096122"} Nov 04 00:58:01.693255 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:01.693183 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vst89" event={"ID":"c6eada63-a80f-4eff-8076-6f01b9efa23e","Type":"ContainerStarted","Data":"2c2ff0ae8b6107ced8dc11426662d452592f20733f317e1f9d573ff3f9f48ddb"} Nov 04 00:58:01.699126 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:01.699083 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8d5qr" event={"ID":"402ccc5f-d755-4061-a96b-8ddc0495b711","Type":"ContainerStarted","Data":"95ffdb116aae4604d44141563ca1e3486d047bf765addbbb136234608ae2b4c4"} Nov 04 00:58:01.709019 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:01.708968 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlx4q" event={"ID":"f784deee-bbd7-4e50-94d1-7bb64fc61b91","Type":"ContainerStarted","Data":"4832f0e32b037011ba261e4a227f95b569e553145a1aec19d3bd81bebceef542"} Nov 04 00:58:01.733017 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:01.732989 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" event={"ID":"847ad3d1-d945-4281-9a5b-15420289c70d","Type":"ContainerStarted","Data":"1ff8d5a10c88fcfb0fa26a96783e8067f5a9915e76562fcbd7995fd130f4b2de"} Nov 04 00:58:01.738149 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:01.738123 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7vrj7" event={"ID":"7cbe8a51-0318-450b-adb2-4abfe1097883","Type":"ContainerStarted","Data":"928b0878ac73f0c4f75f2d9253aa831dae7a8d135a26d976dbd74a8cf4f1f552"} Nov 04 00:58:02.112051 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:02.111961 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:02.112051 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:02.112036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs\") pod \"network-metrics-daemon-dsxl7\" (UID: \"bb2c2fb8-f28c-430b-8210-db7032a7073b\") " pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:02.112328 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:02.112214 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:58:02.112328 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:02.112275 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs podName:bb2c2fb8-f28c-430b-8210-db7032a7073b nodeName:}" failed. No retries permitted until 2026-11-04 00:58:04.112254649 +0000 UTC m=+14.095792511 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs") pod "network-metrics-daemon-dsxl7" (UID: "bb2c2fb8-f28c-430b-8210-db7032a7073b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:58:02.112710 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:02.112685 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:58:02.112799 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:02.112747 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret podName:47fb874a-90bd-41f9-9a8c-0e920d789ca2 nodeName:}" failed. No retries permitted until 2026-11-04 00:58:04.112731265 +0000 UTC m=+14.096269111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret") pod "global-pull-secret-syncer-5v9xx" (UID: "47fb874a-90bd-41f9-9a8c-0e920d789ca2") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:58:02.212815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:02.212775 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llvc6\" (UniqueName: \"kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6\") pod \"network-check-target-b4k7w\" (UID: \"0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec\") " pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:02.213034 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:02.213014 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:58:02.213102 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:02.213042 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:58:02.213102 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:02.213060 2579 projected.go:196] Error preparing data for projected volume kube-api-access-llvc6 for pod openshift-network-diagnostics/network-check-target-b4k7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:58:02.213207 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:02.213141 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6 podName:0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec nodeName:}" failed. No retries permitted until 2026-11-04 00:58:04.213121288 +0000 UTC m=+14.196659155 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-llvc6" (UniqueName: "kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6") pod "network-check-target-b4k7w" (UID: "0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:58:02.441737 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:02.441650 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 00:53:00 +0000 UTC" deadline="2028-04-17 09:28:29.787193921 +0000 UTC" Nov 04 00:58:02.441737 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:02.441689 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12728h30m27.34550914s" Nov 04 00:58:02.583815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:02.583783 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:02.583995 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:02.583915 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9xx" podUID="47fb874a-90bd-41f9-9a8c-0e920d789ca2" Nov 04 00:58:02.586077 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:02.585884 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:02.586077 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:02.586016 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4k7w" podUID="0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec" Nov 04 00:58:02.768058 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:02.767958 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" event={"ID":"f31320b40fb2fb40491617f2b2422d38","Type":"ContainerStarted","Data":"ba6501ed90a71c6188a16ab5392f0e919818e7459cb3ea99966cf5cb49712cf9"} Nov 04 00:58:02.784877 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:02.784816 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" podStartSLOduration=3.784795969 podStartE2EDuration="3.784795969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:02.784391594 +0000 UTC m=+12.767929462" watchObservedRunningTime="2026-11-04 00:58:02.784795969 +0000 UTC m=+12.768333836" Nov 04 00:58:03.582481 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:03.582449 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:03.582697 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:03.582598 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsxl7" podUID="bb2c2fb8-f28c-430b-8210-db7032a7073b" Nov 04 00:58:04.129794 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:04.129764 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:04.130213 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:04.129824 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs\") pod \"network-metrics-daemon-dsxl7\" (UID: \"bb2c2fb8-f28c-430b-8210-db7032a7073b\") " pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:04.130213 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:04.129972 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:58:04.130213 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:04.130026 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs podName:bb2c2fb8-f28c-430b-8210-db7032a7073b nodeName:}" failed. No retries permitted until 2026-11-04 00:58:08.130008536 +0000 UTC m=+18.113546392 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs") pod "network-metrics-daemon-dsxl7" (UID: "bb2c2fb8-f28c-430b-8210-db7032a7073b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:58:04.130478 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:04.130461 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:58:04.130577 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:04.130523 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret podName:47fb874a-90bd-41f9-9a8c-0e920d789ca2 nodeName:}" failed. No retries permitted until 2026-11-04 00:58:08.130508316 +0000 UTC m=+18.114046160 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret") pod "global-pull-secret-syncer-5v9xx" (UID: "47fb874a-90bd-41f9-9a8c-0e920d789ca2") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:58:04.230407 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:04.230334 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llvc6\" (UniqueName: \"kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6\") pod \"network-check-target-b4k7w\" (UID: \"0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec\") " pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:04.230656 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:04.230542 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:58:04.230656 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:04.230579 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:58:04.230656 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:04.230591 2579 projected.go:196] Error preparing data for projected volume kube-api-access-llvc6 for pod openshift-network-diagnostics/network-check-target-b4k7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:58:04.230656 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:04.230645 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6 podName:0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec nodeName:}" failed. No retries permitted until 2026-11-04 00:58:08.230628477 +0000 UTC m=+18.214166333 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-llvc6" (UniqueName: "kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6") pod "network-check-target-b4k7w" (UID: "0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:58:04.583081 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:04.583003 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:04.583250 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:04.583149 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9xx" podUID="47fb874a-90bd-41f9-9a8c-0e920d789ca2" Nov 04 00:58:04.583583 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:04.583546 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:04.583687 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:04.583662 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4k7w" podUID="0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec" Nov 04 00:58:04.781982 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:04.781144 2579 generic.go:356] "Generic (PLEG): container finished" podID="f784deee-bbd7-4e50-94d1-7bb64fc61b91" containerID="5a9107488220c7b7bfa765b47cc3e38765b2d0c8c1ce85297ecbe30fac04d1c0" exitCode=0 Nov 04 00:58:04.781982 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:04.781193 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlx4q" event={"ID":"f784deee-bbd7-4e50-94d1-7bb64fc61b91","Type":"ContainerDied","Data":"5a9107488220c7b7bfa765b47cc3e38765b2d0c8c1ce85297ecbe30fac04d1c0"} Nov 04 00:58:04.992891 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:04.991848 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vqnkh"] Nov 04 00:58:04.996432 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:04.996395 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.022428 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.022405 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 00:58:05.022541 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.022485 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 04 00:58:05.022899 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.022855 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 00:58:05.022984 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.022932 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 04 00:58:05.023494 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.023474 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 04 00:58:05.023645 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.023629 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5llll\"" Nov 04 00:58:05.023725 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.023712 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 04 00:58:05.037062 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.037033 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-tls\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.037143 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.037086 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d50c29d-9104-4dd5-9499-7dd94594666b-sys\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.037143 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.037134 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65cwq\" (UniqueName: \"kubernetes.io/projected/5d50c29d-9104-4dd5-9499-7dd94594666b-kube-api-access-65cwq\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.037253 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.037166 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d50c29d-9104-4dd5-9499-7dd94594666b-metrics-client-ca\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.037253 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.037212 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d50c29d-9104-4dd5-9499-7dd94594666b-root\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.037253 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.037236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-wtmp\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.037385 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.037260 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-accelerators-collector-config\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.037385 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.037291 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-textfile\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.037385 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.037342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.137676 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.137645 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.138049 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.137697 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-tls\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.138049 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.137763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d50c29d-9104-4dd5-9499-7dd94594666b-sys\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.138049 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.137826 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65cwq\" (UniqueName: \"kubernetes.io/projected/5d50c29d-9104-4dd5-9499-7dd94594666b-kube-api-access-65cwq\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.138049 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.137852 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d50c29d-9104-4dd5-9499-7dd94594666b-metrics-client-ca\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.138049 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.137926 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d50c29d-9104-4dd5-9499-7dd94594666b-root\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.138049 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.137952 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-wtmp\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.138049 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.137979 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-accelerators-collector-config\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.138049 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.138008 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-textfile\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.138423 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.138361 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-textfile\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.138897 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.138872 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d50c29d-9104-4dd5-9499-7dd94594666b-sys\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.139024 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.139006 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-wtmp\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.139076 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.139057 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d50c29d-9104-4dd5-9499-7dd94594666b-root\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.139284 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.139258 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d50c29d-9104-4dd5-9499-7dd94594666b-metrics-client-ca\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.139422 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.139405 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-accelerators-collector-config\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.142819 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.142792 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.153953 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.153929 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65cwq\" (UniqueName: \"kubernetes.io/projected/5d50c29d-9104-4dd5-9499-7dd94594666b-kube-api-access-65cwq\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.157684 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.157183 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-tls\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.315672 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.315638 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:05.582519 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.582427 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:05.582694 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:05.582588 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsxl7" podUID="bb2c2fb8-f28c-430b-8210-db7032a7073b" Nov 04 00:58:05.794590 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.794529 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8d5qr" event={"ID":"402ccc5f-d755-4061-a96b-8ddc0495b711","Type":"ContainerStarted","Data":"ddb0f56bea2d4c66c75eef6ef315ad91c667ec25d89c0d1ba4be81ce068ad10e"} Nov 04 00:58:05.799354 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.799322 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vqnkh" event={"ID":"5d50c29d-9104-4dd5-9499-7dd94594666b","Type":"ContainerStarted","Data":"af1da2809571589a0f36763d612938735298294e863cbf70f0576500fab216b5"} Nov 04 00:58:05.922172 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:05.921792 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:06.137418 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:06.137331 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:06.582970 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:06.582810 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:06.582970 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:06.582951 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9xx" podUID="47fb874a-90bd-41f9-9a8c-0e920d789ca2" Nov 04 00:58:06.583657 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:06.583375 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:06.583657 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:06.583476 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4k7w" podUID="0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec" Nov 04 00:58:06.805281 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:06.805241 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v8q47" event={"ID":"565a71a6-1ce7-4078-90fc-f2ccadaa0d17","Type":"ContainerStarted","Data":"490160fd05e79a666322aac2b75f7c6f9e59be52f84664500fd79b91a7fa7af9"} Nov 04 00:58:06.822692 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:06.822648 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8d5qr" podStartSLOduration=5.607334071 podStartE2EDuration="9.822630579s" totalImagesPullingTime="4.215296508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:05.816241918 +0000 UTC m=+15.799779790" watchObservedRunningTime="2026-11-04 00:58:06.822630579 +0000 UTC m=+16.806168440" Nov 04 00:58:06.974340 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:06.974111 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:07.119064 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:07.118997 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:07.582944 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:07.582420 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:07.582944 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:07.582591 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsxl7" podUID="bb2c2fb8-f28c-430b-8210-db7032a7073b" Nov 04 00:58:07.811287 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:07.811209 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" event={"ID":"975903c9-9da2-4971-a50b-7bcc37df6b34","Type":"ContainerStarted","Data":"1cb56badda20ad13cb0edc2e61052b9076c8795869aba40b94d22d246378b7ae"} Nov 04 00:58:07.816437 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:07.816392 2579 generic.go:356] "Generic (PLEG): container finished" podID="5d50c29d-9104-4dd5-9499-7dd94594666b" containerID="fc11b4ec5655633f57e5583270dfceea5e62a822dfdaa0d87fbbcedb6dbc3adc" exitCode=0 Nov 04 00:58:07.816987 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:07.816895 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vqnkh" event={"ID":"5d50c29d-9104-4dd5-9499-7dd94594666b","Type":"ContainerDied","Data":"fc11b4ec5655633f57e5583270dfceea5e62a822dfdaa0d87fbbcedb6dbc3adc"} Nov 04 00:58:07.845817 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:07.845768 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v8q47" podStartSLOduration=5.521540308 podStartE2EDuration="10.845750012s" totalImagesPullingTime="5.324209704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:06.823183593 +0000 UTC m=+16.806721455" watchObservedRunningTime="2026-11-04 00:58:07.845750012 +0000 UTC m=+17.829287874" Nov 04 00:58:07.946779 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:07.946743 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:08.143528 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:08.143493 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:08.163104 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:08.163070 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:08.163251 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:08.163125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs\") pod \"network-metrics-daemon-dsxl7\" (UID: \"bb2c2fb8-f28c-430b-8210-db7032a7073b\") " pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:08.163251 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:08.163246 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:58:08.163364 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:08.163302 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs podName:bb2c2fb8-f28c-430b-8210-db7032a7073b nodeName:}" failed. No retries permitted until 2026-11-04 00:58:16.163284872 +0000 UTC m=+26.146822717 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs") pod "network-metrics-daemon-dsxl7" (UID: "bb2c2fb8-f28c-430b-8210-db7032a7073b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:58:08.163695 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:08.163667 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:58:08.163789 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:08.163717 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret podName:47fb874a-90bd-41f9-9a8c-0e920d789ca2 nodeName:}" failed. No retries permitted until 2026-11-04 00:58:16.163701239 +0000 UTC m=+26.147239097 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret") pod "global-pull-secret-syncer-5v9xx" (UID: "47fb874a-90bd-41f9-9a8c-0e920d789ca2") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:58:08.264118 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:08.264081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llvc6\" (UniqueName: \"kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6\") pod \"network-check-target-b4k7w\" (UID: \"0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec\") " pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:08.264368 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:08.264343 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:58:08.264459 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:08.264377 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:58:08.264459 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:08.264391 2579 projected.go:196] Error preparing data for projected volume kube-api-access-llvc6 for pod openshift-network-diagnostics/network-check-target-b4k7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:58:08.264459 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:08.264449 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6 podName:0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec nodeName:}" failed. No retries permitted until 2026-11-04 00:58:16.264430854 +0000 UTC m=+26.247968862 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-llvc6" (UniqueName: "kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6") pod "network-check-target-b4k7w" (UID: "0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:58:08.339169 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:08.339095 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:08.582812 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:08.582779 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:08.582975 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:08.582901 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9xx" podUID="47fb874a-90bd-41f9-9a8c-0e920d789ca2" Nov 04 00:58:08.583293 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:08.583270 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:08.583840 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:08.583361 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4k7w" podUID="0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec" Nov 04 00:58:08.689474 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:08.689444 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:08.768391 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:08.768363 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 00:58:08.825456 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:08.825359 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" event={"ID":"975903c9-9da2-4971-a50b-7bcc37df6b34","Type":"ContainerStarted","Data":"2c3bdad3286445d0ecbdd9c33c3b7c276f63739aa0859cab69e8bf6ba5f65702"} Nov 04 00:58:08.828911 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:08.828061 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dxbxk" event={"ID":"ac5e9e6b-7405-4c03-a149-4bf353209f24","Type":"ContainerStarted","Data":"d5b317a363d1067c5db5817fe387bc899460d6e62ecd769abd79a4c38c3e1c2e"} Nov 04 00:58:08.833259 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:08.831733 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vqnkh" event={"ID":"5d50c29d-9104-4dd5-9499-7dd94594666b","Type":"ContainerStarted","Data":"62db62b78bdc2537730586763d6b7b77bcf978de892f28f51412bf7b89271ab7"} Nov 04 00:58:08.833259 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:08.831785 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vqnkh" event={"ID":"5d50c29d-9104-4dd5-9499-7dd94594666b","Type":"ContainerStarted","Data":"691868eb59a75ed95dc516158993dda691f314b954305bf6e9a4422c5e40cbc9"} Nov 04 00:58:08.842294 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:08.842247 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:08.853302 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:08.852539 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-dxbxk" podStartSLOduration=3.8807164629999997 podStartE2EDuration="11.852522694s" totalImagesPullingTime="7.971806231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:08.851946166 +0000 UTC m=+18.835484039" watchObservedRunningTime="2026-11-04 00:58:08.852522694 +0000 UTC m=+18.836060555" Nov 04 00:58:08.877004 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:08.876907 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vqnkh" podStartSLOduration=2.6163443429999997 podStartE2EDuration="4.876889349s" totalImagesPullingTime="2.17573653s" totalInitContainerRuntime="84.808476ms" isStatefulPod=true podCreationTimestamp="2026-11-04 00:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:08.875874133 +0000 UTC m=+18.859411995" watchObservedRunningTime="2026-11-04 00:58:08.876889349 +0000 UTC m=+18.860427214" Nov 04 00:58:09.474228 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:09.474119 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T00:58:08.768376236Z","UUID":"af681975-090f-4d67-87f6-72389dba6fe2","Handler":null,"Name":"","Endpoint":""} Nov 04 00:58:09.475722 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:09.475702 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 00:58:09.475796 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:09.475729 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 00:58:09.582505 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:09.582475 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:09.582688 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:09.582642 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsxl7" podUID="bb2c2fb8-f28c-430b-8210-db7032a7073b" Nov 04 00:58:09.623153 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:09.623100 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:09.823482 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:09.823445 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:09.836667 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:09.836636 2579 generic.go:356] "Generic (PLEG): container finished" podID="f784deee-bbd7-4e50-94d1-7bb64fc61b91" containerID="ae8657394b127fdcc8096574f941d34e8a30f83f69afea865a9355275be81005" exitCode=0 Nov 04 00:58:09.836792 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:09.836722 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlx4q" event={"ID":"f784deee-bbd7-4e50-94d1-7bb64fc61b91","Type":"ContainerDied","Data":"ae8657394b127fdcc8096574f941d34e8a30f83f69afea865a9355275be81005"} Nov 04 00:58:09.840082 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:09.840059 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7vrj7" event={"ID":"7cbe8a51-0318-450b-adb2-4abfe1097883","Type":"ContainerStarted","Data":"15ef1b8b2bf2973795682912adfb9adc7b224044e0f302444ffd7df34ec8a575"} Nov 04 00:58:09.965930 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:09.965862 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:10.590137 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:10.590068 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:10.590295 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:10.590183 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9xx" podUID="47fb874a-90bd-41f9-9a8c-0e920d789ca2" Nov 04 00:58:10.590473 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:10.590447 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:10.590549 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:10.590536 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4k7w" podUID="0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec" Nov 04 00:58:10.728306 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:10.728276 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8d5qr" Nov 04 00:58:10.729332 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:10.729125 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8d5qr" Nov 04 00:58:10.748424 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:10.748373 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-7vrj7" podStartSLOduration=5.134530972 podStartE2EDuration="13.748358441s" totalImagesPullingTime="8.613827469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:09.879464284 +0000 UTC m=+19.863002146" watchObservedRunningTime="2026-11-04 00:58:10.748358441 +0000 UTC m=+20.731896303" Nov 04 00:58:10.846428 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:10.846399 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" event={"ID":"975903c9-9da2-4971-a50b-7bcc37df6b34","Type":"ContainerStarted","Data":"bc966edddbdcba3a67992f0e50d3ebb5145af3e80c752d29847819a3868cfc41"} Nov 04 00:58:10.849173 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:10.849153 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jlf57" event={"ID":"f6083261-f4cb-41ee-999d-8c5406473ff8","Type":"ContainerStarted","Data":"e714285e10809f30e0a40d6729c558e49761e75fe03c9f9bef33c5e990ae2a49"} Nov 04 00:58:10.849263 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:10.849191 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8d5qr" Nov 04 00:58:10.850340 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:10.850299 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8d5qr" Nov 04 00:58:10.866068 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:10.865166 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" podStartSLOduration=5.09071968 podStartE2EDuration="13.865148986s" totalImagesPullingTime="8.774429306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:10.865125393 +0000 UTC m=+20.848663255" watchObservedRunningTime="2026-11-04 00:58:10.865148986 +0000 UTC m=+20.848686849" Nov 04 00:58:10.883979 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:10.883939 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jlf57" podStartSLOduration=5.417667538 podStartE2EDuration="13.883926379s" totalImagesPullingTime="8.466258841s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:10.883497541 +0000 UTC m=+20.867035403" watchObservedRunningTime="2026-11-04 00:58:10.883926379 +0000 UTC m=+20.867464243" Nov 04 00:58:11.020327 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:11.020287 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:11.325963 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:11.325929 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:11.582806 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:11.582778 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:11.582970 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:11.582916 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsxl7" podUID="bb2c2fb8-f28c-430b-8210-db7032a7073b" Nov 04 00:58:11.852494 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:11.852415 2579 generic.go:356] "Generic (PLEG): container finished" podID="f784deee-bbd7-4e50-94d1-7bb64fc61b91" containerID="67ec8c3003a1bddffdf49ac5f78b5c1393e9702b83f3014d3d28d8cb79618ee4" exitCode=0 Nov 04 00:58:11.852954 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:11.852504 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlx4q" event={"ID":"f784deee-bbd7-4e50-94d1-7bb64fc61b91","Type":"ContainerDied","Data":"67ec8c3003a1bddffdf49ac5f78b5c1393e9702b83f3014d3d28d8cb79618ee4"} Nov 04 00:58:11.854156 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:11.854078 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" event={"ID":"847ad3d1-d945-4281-9a5b-15420289c70d","Type":"ContainerStarted","Data":"f0e714cfdea15fb34811fb937789d3c66a1c3045618836dc5f9a0ddd2c152d79"} Nov 04 00:58:11.905113 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:11.905066 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" podStartSLOduration=4.488711945 podStartE2EDuration="14.905055656s" totalImagesPullingTime="10.416343711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:11.904344875 +0000 UTC m=+21.887882739" watchObservedRunningTime="2026-11-04 00:58:11.905055656 +0000 UTC m=+21.888593517" Nov 04 00:58:12.582269 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:12.582235 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:12.582543 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:12.582241 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:12.582543 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:12.582360 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9xx" podUID="47fb874a-90bd-41f9-9a8c-0e920d789ca2" Nov 04 00:58:12.582543 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:12.582513 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4k7w" podUID="0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec" Nov 04 00:58:12.859092 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:12.859006 2579 generic.go:356] "Generic (PLEG): container finished" podID="f784deee-bbd7-4e50-94d1-7bb64fc61b91" containerID="4a20ec771a7fc8287cd41f44de4823dcfda6dd91a4cb7f36411c0792b9d3e8f8" exitCode=0 Nov 04 00:58:12.859092 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:12.859048 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlx4q" event={"ID":"f784deee-bbd7-4e50-94d1-7bb64fc61b91","Type":"ContainerDied","Data":"4a20ec771a7fc8287cd41f44de4823dcfda6dd91a4cb7f36411c0792b9d3e8f8"} Nov 04 00:58:13.582311 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:13.582276 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:13.582492 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:13.582418 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsxl7" podUID="bb2c2fb8-f28c-430b-8210-db7032a7073b" Nov 04 00:58:14.582611 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:14.582579 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:14.582611 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:14.582595 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:14.583073 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:14.582691 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4k7w" podUID="0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec" Nov 04 00:58:14.583073 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:14.582771 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9xx" podUID="47fb874a-90bd-41f9-9a8c-0e920d789ca2" Nov 04 00:58:14.867394 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:14.867310 2579 generic.go:356] "Generic (PLEG): container finished" podID="f784deee-bbd7-4e50-94d1-7bb64fc61b91" containerID="8c2d34649e814236c82d3b122c5a1da9680fd3acf1d8e99c8713f9e3761053bc" exitCode=0 Nov 04 00:58:14.867394 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:14.867360 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlx4q" event={"ID":"f784deee-bbd7-4e50-94d1-7bb64fc61b91","Type":"ContainerDied","Data":"8c2d34649e814236c82d3b122c5a1da9680fd3acf1d8e99c8713f9e3761053bc"} Nov 04 00:58:15.582001 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:15.581966 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:15.582176 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:15.582105 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsxl7" podUID="bb2c2fb8-f28c-430b-8210-db7032a7073b" Nov 04 00:58:15.872256 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:15.872233 2579 generic.go:356] "Generic (PLEG): container finished" podID="f784deee-bbd7-4e50-94d1-7bb64fc61b91" containerID="dfd9c596718db5027553ea47a0f1b27cd0deb96690660fa8e3f2e99be8f4683c" exitCode=0 Nov 04 00:58:15.872647 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:15.872276 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlx4q" event={"ID":"f784deee-bbd7-4e50-94d1-7bb64fc61b91","Type":"ContainerDied","Data":"dfd9c596718db5027553ea47a0f1b27cd0deb96690660fa8e3f2e99be8f4683c"} Nov 04 00:58:16.230731 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:16.230580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs\") pod \"network-metrics-daemon-dsxl7\" (UID: \"bb2c2fb8-f28c-430b-8210-db7032a7073b\") " pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:16.230815 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:16.230774 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:16.230815 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:16.230649 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:58:16.230887 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:16.230842 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs podName:bb2c2fb8-f28c-430b-8210-db7032a7073b nodeName:}" failed. No retries permitted until 2026-11-04 00:58:32.230829051 +0000 UTC m=+42.214366894 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs") pod "network-metrics-daemon-dsxl7" (UID: "bb2c2fb8-f28c-430b-8210-db7032a7073b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:58:16.230887 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:16.230847 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:58:16.230887 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:16.230879 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret podName:47fb874a-90bd-41f9-9a8c-0e920d789ca2 nodeName:}" failed. No retries permitted until 2026-11-04 00:58:32.230869154 +0000 UTC m=+42.214407009 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret") pod "global-pull-secret-syncer-5v9xx" (UID: "47fb874a-90bd-41f9-9a8c-0e920d789ca2") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:58:16.331438 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:16.331413 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llvc6\" (UniqueName: \"kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6\") pod \"network-check-target-b4k7w\" (UID: \"0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec\") " pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:16.331587 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:16.331537 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:58:16.331587 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:16.331550 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:58:16.331587 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:16.331580 2579 projected.go:196] Error preparing data for projected volume kube-api-access-llvc6 for pod openshift-network-diagnostics/network-check-target-b4k7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:58:16.331683 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:16.331629 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6 podName:0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec nodeName:}" failed. No retries permitted until 2026-11-04 00:58:32.331614414 +0000 UTC m=+42.315152256 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-llvc6" (UniqueName: "kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6") pod "network-check-target-b4k7w" (UID: "0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:58:16.582130 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:16.582102 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:16.582257 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:16.582219 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4k7w" podUID="0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec" Nov 04 00:58:16.582296 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:16.582271 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:16.582378 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:16.582361 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9xx" podUID="47fb874a-90bd-41f9-9a8c-0e920d789ca2" Nov 04 00:58:16.876796 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:16.876733 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 00:58:16.877433 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:16.877105 2579 generic.go:356] "Generic (PLEG): container finished" podID="c6eada63-a80f-4eff-8076-6f01b9efa23e" containerID="21ce13a10350e0cfc715625dc822391a209874e65b247efeca0d556a99a48bcb" exitCode=1 Nov 04 00:58:16.877433 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:16.877185 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vst89" event={"ID":"c6eada63-a80f-4eff-8076-6f01b9efa23e","Type":"ContainerStarted","Data":"8184d06f05bb1f7aa6f08c46c520c9c0531094e74510fe842fdaccc26cab810d"} Nov 04 00:58:16.877433 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:16.877265 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vst89" event={"ID":"c6eada63-a80f-4eff-8076-6f01b9efa23e","Type":"ContainerStarted","Data":"fd4c37b5504445e7349ede5c7979981828a5b319da6a1d8365e2042a24f1c2cf"} Nov 04 00:58:16.877433 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:16.877313 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vst89" event={"ID":"c6eada63-a80f-4eff-8076-6f01b9efa23e","Type":"ContainerStarted","Data":"9506384c6fdaf40db391de08840af393d33967192058abb4dc60e35705c85522"} Nov 04 00:58:16.877433 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:16.877356 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vst89" event={"ID":"c6eada63-a80f-4eff-8076-6f01b9efa23e","Type":"ContainerStarted","Data":"974d3745d0a2bc4ed145dbce121b297b7dad50061074c3ccee2642d5737544e2"} Nov 04 00:58:16.877433 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:16.877397 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vst89" event={"ID":"c6eada63-a80f-4eff-8076-6f01b9efa23e","Type":"ContainerDied","Data":"21ce13a10350e0cfc715625dc822391a209874e65b247efeca0d556a99a48bcb"} Nov 04 00:58:16.877673 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:16.877443 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vst89" event={"ID":"c6eada63-a80f-4eff-8076-6f01b9efa23e","Type":"ContainerStarted","Data":"3989daa501f9e5d42c1f0fed2311a57d1d90e9cb727eff1bf3fef4756bf0e843"} Nov 04 00:58:16.880081 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:16.880057 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlx4q" event={"ID":"f784deee-bbd7-4e50-94d1-7bb64fc61b91","Type":"ContainerStarted","Data":"797627283a2084654dbd1dff52aaed446e30b72363527d1568e20687cc9846fe"} Nov 04 00:58:16.902081 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:16.902041 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mlx4q" podStartSLOduration=8.788756535 podStartE2EDuration="19.902025672s" totalImagesPullingTime="11.113269137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:16.901714898 +0000 UTC m=+26.885252760" watchObservedRunningTime="2026-11-04 00:58:16.902025672 +0000 UTC m=+26.885563535" Nov 04 00:58:17.582870 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:17.582841 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:17.583019 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:17.582942 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsxl7" podUID="bb2c2fb8-f28c-430b-8210-db7032a7073b" Nov 04 00:58:17.615458 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:17.615430 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:17.743261 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:17.743235 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:18.355831 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:18.355795 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:18.498121 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:18.498090 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:18.582187 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:18.582133 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:18.582286 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:18.582260 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:18.582286 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:18.582259 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4k7w" podUID="0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec" Nov 04 00:58:18.582363 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:18.582332 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9xx" podUID="47fb874a-90bd-41f9-9a8c-0e920d789ca2" Nov 04 00:58:18.886687 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:18.886670 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 00:58:18.887030 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:18.887002 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vst89" event={"ID":"c6eada63-a80f-4eff-8076-6f01b9efa23e","Type":"ContainerStarted","Data":"789b935b2465c675265042464026fc41e644549aa2f262edfcbff7a0851ad6fe"} Nov 04 00:58:19.051820 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:19.051792 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:19.180108 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:19.180018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:19.582184 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:19.582157 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:19.582677 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:19.582258 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsxl7" podUID="bb2c2fb8-f28c-430b-8210-db7032a7073b" Nov 04 00:58:20.585558 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:20.585390 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:20.585558 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:20.585397 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:20.586127 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:20.585650 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9xx" podUID="47fb874a-90bd-41f9-9a8c-0e920d789ca2" Nov 04 00:58:20.586127 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:20.585751 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4k7w" podUID="0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec" Nov 04 00:58:20.893403 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:20.893378 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 00:58:20.893748 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:20.893727 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vst89" event={"ID":"c6eada63-a80f-4eff-8076-6f01b9efa23e","Type":"ContainerStarted","Data":"53de2740762f4df9a0ff7ca7d04cdfe2cdbfc34a6a7334f5e743449b7106f4bc"} Nov 04 00:58:20.893971 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:20.893955 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:20.894076 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:20.894062 2579 scope.go:122] "RemoveContainer" containerID="21ce13a10350e0cfc715625dc822391a209874e65b247efeca0d556a99a48bcb" Nov 04 00:58:20.911613 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:20.911555 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:21.581919 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:21.581892 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:21.582150 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:21.582020 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsxl7" podUID="bb2c2fb8-f28c-430b-8210-db7032a7073b" Nov 04 00:58:21.900296 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:21.900275 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 00:58:21.900695 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:21.900675 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vst89" event={"ID":"c6eada63-a80f-4eff-8076-6f01b9efa23e","Type":"ContainerStarted","Data":"6a757e0e3bf95d77a88ad091372095f1e1317fd3dfc2c7d7e7f36aab1fa54d36"} Nov 04 00:58:21.900778 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:21.900765 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:58:21.901020 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:21.901008 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:21.916071 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:21.916046 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:21.944004 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:21.943969 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vst89" podStartSLOduration=9.82103631 podStartE2EDuration="24.943957159s" totalImagesPullingTime="15.122920849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:21.943537696 +0000 UTC m=+31.927075557" watchObservedRunningTime="2026-11-04 00:58:21.943957159 +0000 UTC m=+31.927495020" Nov 04 00:58:22.141006 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:22.140937 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b4k7w"] Nov 04 00:58:22.141131 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:22.141073 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:22.141211 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:22.141189 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4k7w" podUID="0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec" Nov 04 00:58:22.145527 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:22.145500 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dsxl7"] Nov 04 00:58:22.145655 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:22.145587 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5v9xx"] Nov 04 00:58:22.145655 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:22.145649 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:22.145726 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:22.145717 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9xx" podUID="47fb874a-90bd-41f9-9a8c-0e920d789ca2" Nov 04 00:58:22.145769 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:22.145758 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:22.145822 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:22.145811 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsxl7" podUID="bb2c2fb8-f28c-430b-8210-db7032a7073b" Nov 04 00:58:22.902718 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:22.902693 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:58:23.582797 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:23.582765 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:23.582797 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:23.582795 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:23.583000 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:23.582765 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:23.583000 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:23.582858 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9xx" podUID="47fb874a-90bd-41f9-9a8c-0e920d789ca2" Nov 04 00:58:23.583000 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:23.582944 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4k7w" podUID="0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec" Nov 04 00:58:23.583098 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:23.583046 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsxl7" podUID="bb2c2fb8-f28c-430b-8210-db7032a7073b" Nov 04 00:58:23.904703 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:23.904638 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:58:25.582577 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:25.582527 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:25.582577 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:25.582528 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:25.583001 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:25.582527 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:25.583001 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:25.582683 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsxl7" podUID="bb2c2fb8-f28c-430b-8210-db7032a7073b" Nov 04 00:58:25.583001 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:25.582787 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9xx" podUID="47fb874a-90bd-41f9-9a8c-0e920d789ca2" Nov 04 00:58:25.583001 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:25.582843 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4k7w" podUID="0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec" Nov 04 00:58:26.939366 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:26.939183 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:27.141765 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:27.141718 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:27.339936 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:27.339910 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:27.582498 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:27.582471 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:27.582498 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:27.582494 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:27.582728 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:27.582519 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:27.582728 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:27.582618 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4k7w" podUID="0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec" Nov 04 00:58:27.582728 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:27.582686 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsxl7" podUID="bb2c2fb8-f28c-430b-8210-db7032a7073b" Nov 04 00:58:27.582851 ip-10-0-140-241 kubenswrapper[2579]: E1104 00:58:27.582727 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9xx" podUID="47fb874a-90bd-41f9-9a8c-0e920d789ca2" Nov 04 00:58:28.277425 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.277400 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeReady" Nov 04 00:58:28.277757 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.277510 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 04 00:58:28.354009 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.353954 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xw58g"] Nov 04 00:58:28.367105 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.367083 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z9szb"] Nov 04 00:58:28.367208 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.367191 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xw58g" Nov 04 00:58:28.379618 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.379597 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xw58g"] Nov 04 00:58:28.379697 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.379678 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:28.382793 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.382773 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pb69q\"" Nov 04 00:58:28.382884 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.382801 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 00:58:28.385908 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.385881 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 00:58:28.385996 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.385956 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 00:58:28.389221 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.389201 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z9szb"] Nov 04 00:58:28.424996 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.424973 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vt9h\" (UniqueName: \"kubernetes.io/projected/66117eac-ae66-462e-8282-7ef04b047fbe-kube-api-access-8vt9h\") pod \"ingress-canary-xw58g\" (UID: \"66117eac-ae66-462e-8282-7ef04b047fbe\") " pod="openshift-ingress-canary/ingress-canary-xw58g" Nov 04 00:58:28.425221 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.425189 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66117eac-ae66-462e-8282-7ef04b047fbe-cert\") pod \"ingress-canary-xw58g\" (UID: \"66117eac-ae66-462e-8282-7ef04b047fbe\") " pod="openshift-ingress-canary/ingress-canary-xw58g" Nov 04 00:58:28.425823 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.425803 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:28.462343 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.462319 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-n2w4k"] Nov 04 00:58:28.476817 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.476792 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:28.485754 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.485736 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cjss9\"" Nov 04 00:58:28.485754 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.485736 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 00:58:28.485903 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.485877 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 00:58:28.485986 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.485971 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 00:58:28.486180 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.486161 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 00:58:28.487018 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.486987 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 00:58:28.487144 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.487128 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ldjm6\"" Nov 04 00:58:28.487213 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.487133 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 00:58:28.487852 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.487833 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n2w4k"] Nov 04 00:58:28.526138 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.526117 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-config-volume\") pod \"dns-default-z9szb\" (UID: \"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c\") " pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:28.526226 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.526150 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-metrics-tls\") pod \"dns-default-z9szb\" (UID: \"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c\") " pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:28.526226 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.526181 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vt9h\" (UniqueName: \"kubernetes.io/projected/66117eac-ae66-462e-8282-7ef04b047fbe-kube-api-access-8vt9h\") pod \"ingress-canary-xw58g\" (UID: \"66117eac-ae66-462e-8282-7ef04b047fbe\") " pod="openshift-ingress-canary/ingress-canary-xw58g" Nov 04 00:58:28.526351 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.526283 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drqhw\" (UniqueName: \"kubernetes.io/projected/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-kube-api-access-drqhw\") pod \"dns-default-z9szb\" (UID: \"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c\") " pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:28.526408 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.526354 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-tmp-dir\") pod \"dns-default-z9szb\" (UID: \"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c\") " pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:28.526408 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.526393 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66117eac-ae66-462e-8282-7ef04b047fbe-cert\") pod \"ingress-canary-xw58g\" (UID: \"66117eac-ae66-462e-8282-7ef04b047fbe\") " pod="openshift-ingress-canary/ingress-canary-xw58g" Nov 04 00:58:28.530345 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.530328 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66117eac-ae66-462e-8282-7ef04b047fbe-cert\") pod \"ingress-canary-xw58g\" (UID: \"66117eac-ae66-462e-8282-7ef04b047fbe\") " pod="openshift-ingress-canary/ingress-canary-xw58g" Nov 04 00:58:28.539232 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.539208 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vt9h\" (UniqueName: \"kubernetes.io/projected/66117eac-ae66-462e-8282-7ef04b047fbe-kube-api-access-8vt9h\") pod \"ingress-canary-xw58g\" (UID: \"66117eac-ae66-462e-8282-7ef04b047fbe\") " pod="openshift-ingress-canary/ingress-canary-xw58g" Nov 04 00:58:28.627652 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.627605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-metrics-tls\") pod \"dns-default-z9szb\" (UID: \"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c\") " pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:28.627766 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.627637 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55e1fd53-92c2-484e-9405-ac5af81ed61b-data-volume\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:28.627766 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.627679 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55e1fd53-92c2-484e-9405-ac5af81ed61b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:28.627766 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.627696 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55e1fd53-92c2-484e-9405-ac5af81ed61b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:28.627766 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.627746 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8q28\" (UniqueName: \"kubernetes.io/projected/55e1fd53-92c2-484e-9405-ac5af81ed61b-kube-api-access-m8q28\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:28.627910 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.627800 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drqhw\" (UniqueName: \"kubernetes.io/projected/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-kube-api-access-drqhw\") pod \"dns-default-z9szb\" (UID: \"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c\") " pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:28.627910 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.627842 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-tmp-dir\") pod \"dns-default-z9szb\" (UID: \"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c\") " pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:28.628064 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.627907 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55e1fd53-92c2-484e-9405-ac5af81ed61b-crio-socket\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:28.628345 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.628257 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-config-volume\") pod \"dns-default-z9szb\" (UID: \"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c\") " pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:28.628824 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.628615 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-tmp-dir\") pod \"dns-default-z9szb\" (UID: \"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c\") " pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:28.631593 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.629198 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-config-volume\") pod \"dns-default-z9szb\" (UID: \"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c\") " pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:28.631593 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.630617 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-metrics-tls\") pod \"dns-default-z9szb\" (UID: \"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c\") " pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:28.639138 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.639120 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drqhw\" (UniqueName: \"kubernetes.io/projected/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-kube-api-access-drqhw\") pod \"dns-default-z9szb\" (UID: \"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c\") " pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:28.686731 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.686709 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xw58g" Nov 04 00:58:28.701181 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.701158 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:28.715313 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.715297 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:28.728628 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.728608 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55e1fd53-92c2-484e-9405-ac5af81ed61b-crio-socket\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:28.728736 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.728653 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55e1fd53-92c2-484e-9405-ac5af81ed61b-data-volume\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:28.728736 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.728672 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55e1fd53-92c2-484e-9405-ac5af81ed61b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:28.728736 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.728688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55e1fd53-92c2-484e-9405-ac5af81ed61b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:28.728736 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.728705 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8q28\" (UniqueName: \"kubernetes.io/projected/55e1fd53-92c2-484e-9405-ac5af81ed61b-kube-api-access-m8q28\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:28.728968 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.728837 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55e1fd53-92c2-484e-9405-ac5af81ed61b-crio-socket\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:28.729022 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.728991 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55e1fd53-92c2-484e-9405-ac5af81ed61b-data-volume\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:28.729331 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.729310 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55e1fd53-92c2-484e-9405-ac5af81ed61b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:28.731174 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.731141 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55e1fd53-92c2-484e-9405-ac5af81ed61b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:28.756526 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.756503 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8q28\" (UniqueName: \"kubernetes.io/projected/55e1fd53-92c2-484e-9405-ac5af81ed61b-kube-api-access-m8q28\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:28.808403 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:28.808373 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:29.019135 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:29.019038 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xw58g"] Nov 04 00:58:29.021768 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:58:29.021747 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66117eac_ae66_462e_8282_7ef04b047fbe.slice/crio-c5c23d2cad0687c62de401a72caf2927884cc276b0be1f6d294366179128eaa7 WatchSource:0}: Error finding container c5c23d2cad0687c62de401a72caf2927884cc276b0be1f6d294366179128eaa7: Status 404 returned error can't find the container with id c5c23d2cad0687c62de401a72caf2927884cc276b0be1f6d294366179128eaa7 Nov 04 00:58:29.024602 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:29.024581 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z9szb"] Nov 04 00:58:29.027538 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:58:29.027518 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a0d5c94_5bd3_4834_9e5e_5ebe04fd979c.slice/crio-97ca9fe8c890a540b9403fd94faf0edb071532c8f5e838a5a6b274bb9ed59f51 WatchSource:0}: Error finding container 97ca9fe8c890a540b9403fd94faf0edb071532c8f5e838a5a6b274bb9ed59f51: Status 404 returned error can't find the container with id 97ca9fe8c890a540b9403fd94faf0edb071532c8f5e838a5a6b274bb9ed59f51 Nov 04 00:58:29.094443 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:29.094415 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n2w4k"] Nov 04 00:58:29.096770 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:58:29.096750 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e1fd53_92c2_484e_9405_ac5af81ed61b.slice/crio-8c348450f2fa7f6112786c73979ebec84d5fbc267418d8b91e15825916c9eefd WatchSource:0}: Error finding container 8c348450f2fa7f6112786c73979ebec84d5fbc267418d8b91e15825916c9eefd: Status 404 returned error can't find the container with id 8c348450f2fa7f6112786c73979ebec84d5fbc267418d8b91e15825916c9eefd Nov 04 00:58:29.582453 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:29.582292 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:29.583157 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:29.582369 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:29.583157 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:29.582376 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:29.587224 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:29.587204 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 00:58:29.592402 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:29.592386 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nszlw\"" Nov 04 00:58:29.592497 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:29.592434 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 00:58:29.594110 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:29.594083 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q48lg\"" Nov 04 00:58:29.594176 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:29.594168 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4rmxc\"" Nov 04 00:58:29.685445 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:29.685421 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:29.687923 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:29.687902 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 00:58:29.693041 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:29.693026 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 00:58:29.920727 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:29.920597 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z9szb" event={"ID":"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c","Type":"ContainerStarted","Data":"97ca9fe8c890a540b9403fd94faf0edb071532c8f5e838a5a6b274bb9ed59f51"} Nov 04 00:58:29.922384 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:29.922354 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xw58g" event={"ID":"66117eac-ae66-462e-8282-7ef04b047fbe","Type":"ContainerStarted","Data":"c5c23d2cad0687c62de401a72caf2927884cc276b0be1f6d294366179128eaa7"} Nov 04 00:58:29.924808 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:29.924766 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n2w4k" event={"ID":"55e1fd53-92c2-484e-9405-ac5af81ed61b","Type":"ContainerStarted","Data":"15b2eec4abd3e174d564c1a5ab69b3689da27054adeb900f891e72b11f4fa14c"} Nov 04 00:58:29.924919 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:29.924818 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n2w4k" event={"ID":"55e1fd53-92c2-484e-9405-ac5af81ed61b","Type":"ContainerStarted","Data":"8c348450f2fa7f6112786c73979ebec84d5fbc267418d8b91e15825916c9eefd"} Nov 04 00:58:29.943304 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:29.943275 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:30.190097 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:30.190029 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:30.190281 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:30.190235 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:58:30.208337 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:30.208313 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:30.277763 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:30.277697 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:30.471915 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:30.471876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:30.929657 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:30.929579 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n2w4k" event={"ID":"55e1fd53-92c2-484e-9405-ac5af81ed61b","Type":"ContainerStarted","Data":"459c825136ab6ad5dbf92d7766588b9dd350e97d23ef7340390b52715ee75d58"} Nov 04 00:58:31.934829 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:31.934778 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z9szb" event={"ID":"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c","Type":"ContainerStarted","Data":"a6c718fe09eb38229eb7c2af57d2cbc650efc01a54cbe4dd483e69526fdd3480"} Nov 04 00:58:32.258781 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:32.258744 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs\") pod \"network-metrics-daemon-dsxl7\" (UID: \"bb2c2fb8-f28c-430b-8210-db7032a7073b\") " pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:32.258985 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:32.258858 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:32.262480 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:32.262427 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:32.262480 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:32.262473 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs\") pod \"network-metrics-daemon-dsxl7\" (UID: \"bb2c2fb8-f28c-430b-8210-db7032a7073b\") " pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:32.302484 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:32.302459 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:32.341686 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:32.341665 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:32.359752 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:32.359731 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llvc6\" (UniqueName: \"kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6\") pod \"network-check-target-b4k7w\" (UID: \"0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec\") " pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:32.362133 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:32.362104 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llvc6\" (UniqueName: \"kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6\") pod \"network-check-target-b4k7w\" (UID: \"0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec\") " pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:32.615496 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:32.615427 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:32.661800 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:32.661775 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5v9xx"] Nov 04 00:58:32.664528 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:32.664507 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dsxl7"] Nov 04 00:58:32.727217 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:58:32.727094 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47fb874a_90bd_41f9_9a8c_0e920d789ca2.slice/crio-9b0240716b1410bacd943746df2af03430f99980a3e0acc79aa62c401898a250 WatchSource:0}: Error finding container 9b0240716b1410bacd943746df2af03430f99980a3e0acc79aa62c401898a250: Status 404 returned error can't find the container with id 9b0240716b1410bacd943746df2af03430f99980a3e0acc79aa62c401898a250 Nov 04 00:58:32.727939 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:58:32.727921 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2c2fb8_f28c_430b_8210_db7032a7073b.slice/crio-277ddfb0de2eee6d0a584457c024d05cb0c5fa9ced843e754720beae14816406 WatchSource:0}: Error finding container 277ddfb0de2eee6d0a584457c024d05cb0c5fa9ced843e754720beae14816406: Status 404 returned error can't find the container with id 277ddfb0de2eee6d0a584457c024d05cb0c5fa9ced843e754720beae14816406 Nov 04 00:58:32.939659 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:32.939577 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z9szb" event={"ID":"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c","Type":"ContainerStarted","Data":"e89ec05b155402abb8ff21c2a270a9e914d4945e9ecd23472f51c78356a938e0"} Nov 04 00:58:32.940059 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:32.939689 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:32.941216 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:32.941187 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xw58g" event={"ID":"66117eac-ae66-462e-8282-7ef04b047fbe","Type":"ContainerStarted","Data":"39a82a04601f732b1a33137e2ad7b93f4d847752fade55bd1cfea125169acafa"} Nov 04 00:58:32.943181 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:32.943154 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n2w4k" event={"ID":"55e1fd53-92c2-484e-9405-ac5af81ed61b","Type":"ContainerStarted","Data":"feec81717c3b4f298f8a1c760962c005006d98654b41dcd430109ab204d74bfe"} Nov 04 00:58:32.944211 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:32.944192 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5v9xx" event={"ID":"47fb874a-90bd-41f9-9a8c-0e920d789ca2","Type":"ContainerStarted","Data":"9b0240716b1410bacd943746df2af03430f99980a3e0acc79aa62c401898a250"} Nov 04 00:58:32.945310 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:32.945289 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dsxl7" event={"ID":"bb2c2fb8-f28c-430b-8210-db7032a7073b","Type":"ContainerStarted","Data":"277ddfb0de2eee6d0a584457c024d05cb0c5fa9ced843e754720beae14816406"} Nov 04 00:58:32.957285 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:32.957216 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-z9szb" podStartSLOduration=2.27676872 podStartE2EDuration="4.957201384s" totalImagesPullingTime="2.680432664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:58:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:32.95711676 +0000 UTC m=+42.940654620" watchObservedRunningTime="2026-11-04 00:58:32.957201384 +0000 UTC m=+42.940739246" Nov 04 00:58:32.973386 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:32.973346 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xw58g" podStartSLOduration=1.2277087660000001 podStartE2EDuration="4.973334798s" totalImagesPullingTime="3.745626032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:58:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:32.97300702 +0000 UTC m=+42.956544881" watchObservedRunningTime="2026-11-04 00:58:32.973334798 +0000 UTC m=+42.956872658" Nov 04 00:58:32.990488 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:32.990450 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-n2w4k" podStartSLOduration=1.522156921 podStartE2EDuration="4.990436947s" totalImagesPullingTime="3.468280026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:58:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:32.989946498 +0000 UTC m=+42.973484362" watchObservedRunningTime="2026-11-04 00:58:32.990436947 +0000 UTC m=+42.973974808" Nov 04 00:58:33.036178 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:33.036151 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b4k7w"] Nov 04 00:58:33.041920 ip-10-0-140-241 kubenswrapper[2579]: W1104 00:58:33.041897 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bfe7ded_d69c_49a5_9d70_b8aa3067e4ec.slice/crio-3fc4c1e4b8b2fb772b8257e1316188ed74a737fd023c3acf4d90b68c20be30c8 WatchSource:0}: Error finding container 3fc4c1e4b8b2fb772b8257e1316188ed74a737fd023c3acf4d90b68c20be30c8: Status 404 returned error can't find the container with id 3fc4c1e4b8b2fb772b8257e1316188ed74a737fd023c3acf4d90b68c20be30c8 Nov 04 00:58:33.951040 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:33.950990 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b4k7w" event={"ID":"0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec","Type":"ContainerStarted","Data":"3fc4c1e4b8b2fb772b8257e1316188ed74a737fd023c3acf4d90b68c20be30c8"} Nov 04 00:58:34.956845 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:34.956804 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dsxl7" event={"ID":"bb2c2fb8-f28c-430b-8210-db7032a7073b","Type":"ContainerStarted","Data":"75db7171e35f5efc7e22ae2b80659e9d3fe00beb2c3be5a4c736c1c826dcb2f1"} Nov 04 00:58:34.957308 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:34.956870 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dsxl7" event={"ID":"bb2c2fb8-f28c-430b-8210-db7032a7073b","Type":"ContainerStarted","Data":"e5fbd5acaecf05834b9edec1e6c8487df0fe99c8166b5cdf5a4aba03cff70efb"} Nov 04 00:58:34.974130 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:34.973520 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dsxl7" podStartSLOduration=36.787220479 podStartE2EDuration="37.973503191s" totalImagesPullingTime="1.186282712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:34.972750742 +0000 UTC m=+44.956288604" watchObservedRunningTime="2026-11-04 00:58:34.973503191 +0000 UTC m=+44.957041055" Nov 04 00:58:36.966426 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:36.966381 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b4k7w" event={"ID":"0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec","Type":"ContainerStarted","Data":"fb983e33b481804af62565b3d652c779d342755704240a1c1e9fbe82e184172c"} Nov 04 00:58:36.966902 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:36.966477 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:36.982464 ip-10-0-140-241 kubenswrapper[2579]: I1104 00:58:36.982419 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b4k7w" podStartSLOduration=36.220456496 podStartE2EDuration="39.982398471s" totalImagesPullingTime="3.761941975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:36.981975677 +0000 UTC m=+46.965513542" watchObservedRunningTime="2026-11-04 00:58:36.982398471 +0000 UTC m=+46.965936332" Nov 04 00:58:37.830792 ip-10-0-140-241 systemd[1]: Stopping Kubernetes Kubelet... Nov 04 00:58:37.839030 ip-10-0-140-241 systemd[1]: kubelet.service: Deactivated successfully. Nov 04 00:58:37.839196 ip-10-0-140-241 systemd[1]: Stopped Kubernetes Kubelet. Nov 04 00:58:37.839359 ip-10-0-140-241 systemd[1]: kubelet.service: Consumed 2.674s CPU time. Nov 04 00:58:37.853243 ip-10-0-140-241 systemd[1]: Starting Kubernetes Kubelet... Nov 04 00:58:37.912473 ip-10-0-140-241 kubenswrapper[4852]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:58:37.912473 ip-10-0-140-241 kubenswrapper[4852]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 00:58:37.912473 ip-10-0-140-241 kubenswrapper[4852]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:58:37.912933 ip-10-0-140-241 kubenswrapper[4852]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:58:37.915380 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915310 4852 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:58:37.915380 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915379 4852 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:58:37.915380 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915383 4852 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:58:37.915509 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915385 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:58:37.915509 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915389 4852 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:58:37.915509 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915391 4852 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:58:37.915509 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915394 4852 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:58:37.915509 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915397 4852 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:58:37.915509 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915399 4852 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:58:37.915509 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915402 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:58:37.915509 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915404 4852 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:58:37.915509 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915408 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:58:37.915509 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915411 4852 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:58:37.915509 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915413 4852 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:58:37.915509 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915416 4852 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:58:37.915509 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915419 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:58:37.915509 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915421 4852 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:58:37.915509 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915424 4852 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:58:37.915509 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915426 4852 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:58:37.915509 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915428 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:58:37.915509 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915431 4852 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:58:37.915509 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915434 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:58:37.915985 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915437 4852 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:58:37.915985 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915439 4852 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:58:37.915985 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915441 4852 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:58:37.915985 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915444 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:58:37.915985 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915446 4852 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:58:37.915985 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915449 4852 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:58:37.915985 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915452 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:58:37.915985 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915455 4852 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:58:37.915985 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915457 4852 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:58:37.915985 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915460 4852 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:58:37.915985 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915467 4852 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:58:37.915985 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915470 4852 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:58:37.915985 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915472 4852 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:58:37.915985 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915475 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:58:37.915985 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915477 4852 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:58:37.915985 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915480 4852 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:58:37.915985 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915483 4852 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:58:37.915985 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915485 4852 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:58:37.915985 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915487 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:58:37.915985 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915490 4852 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:58:37.916590 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915493 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:58:37.916590 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915496 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:58:37.916590 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915498 4852 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:58:37.916590 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915501 4852 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:58:37.916590 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915503 4852 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:58:37.916590 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915506 4852 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:58:37.916590 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915508 4852 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:58:37.916590 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915511 4852 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:58:37.916590 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915514 4852 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:58:37.916590 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915518 4852 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:58:37.916590 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915522 4852 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:58:37.916590 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915525 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:58:37.916590 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915528 4852 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:58:37.916590 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915530 4852 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:58:37.916590 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915533 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:58:37.916590 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915535 4852 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:58:37.916590 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915538 4852 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:58:37.916590 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915542 4852 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:58:37.916590 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915544 4852 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:58:37.917043 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915547 4852 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:58:37.917043 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915550 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:58:37.917043 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915552 4852 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:58:37.917043 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915555 4852 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:58:37.917043 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915577 4852 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:58:37.917043 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915580 4852 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:58:37.917043 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915583 4852 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:58:37.917043 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915585 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:58:37.917043 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915588 4852 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:58:37.917043 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915591 4852 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:58:37.917043 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915593 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:58:37.917043 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915596 4852 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:58:37.917043 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915600 4852 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:58:37.917043 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915604 4852 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:58:37.917043 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915607 4852 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:58:37.917043 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915610 4852 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:58:37.917043 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915613 4852 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:58:37.917043 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915616 4852 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:58:37.917043 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915618 4852 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:58:37.917043 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915620 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:58:37.917578 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915623 4852 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:58:37.917578 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915625 4852 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:58:37.917578 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915629 4852 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:58:37.917578 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915631 4852 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:58:37.917578 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915634 4852 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:58:37.917578 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915637 4852 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:58:37.917578 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915639 4852 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:58:37.917578 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915642 4852 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:58:37.917578 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915644 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:58:37.917578 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915647 4852 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:58:37.917578 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915650 4852 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:58:37.917578 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915652 4852 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:58:37.917578 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915655 4852 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:58:37.917578 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915657 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:58:37.917578 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915660 4852 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:58:37.917578 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915662 4852 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:58:37.917578 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915665 4852 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:58:37.917578 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915667 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:58:37.917578 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915670 4852 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:58:37.917578 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915672 4852 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:58:37.918053 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915674 4852 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:58:37.918053 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915677 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:58:37.918053 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915679 4852 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:58:37.918053 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915683 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:58:37.918053 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.915685 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:58:37.918053 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916416 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:58:37.918053 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916422 4852 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:58:37.918053 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916425 4852 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:58:37.918053 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916427 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:58:37.918053 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916430 4852 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:58:37.918053 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916433 4852 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:58:37.918053 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916435 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:58:37.918053 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916438 4852 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:58:37.918053 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916440 4852 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:58:37.918053 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916443 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:58:37.918053 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916446 4852 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:58:37.918053 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916449 4852 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:58:37.918053 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916452 4852 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:58:37.918053 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916455 4852 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:58:37.918523 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916458 4852 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:58:37.918523 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916460 4852 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:58:37.918523 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916464 4852 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:58:37.918523 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916466 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:58:37.918523 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916469 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:58:37.918523 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916471 4852 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:58:37.918523 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916474 4852 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:58:37.918523 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916476 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:58:37.918523 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916479 4852 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:58:37.918523 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916481 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:58:37.918523 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916484 4852 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:58:37.918523 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916486 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:58:37.918523 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916489 4852 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:58:37.918523 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916491 4852 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:58:37.918523 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916494 4852 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:58:37.918523 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916496 4852 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:58:37.918523 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916498 4852 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:58:37.918523 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916501 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:58:37.918523 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916503 4852 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:58:37.919021 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916506 4852 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:58:37.919021 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916508 4852 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:58:37.919021 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916511 4852 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:58:37.919021 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916513 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:58:37.919021 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916516 4852 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:58:37.919021 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916519 4852 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:58:37.919021 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916521 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:58:37.919021 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916523 4852 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:58:37.919021 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916526 4852 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:58:37.919021 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916529 4852 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:58:37.919021 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916531 4852 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:58:37.919021 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916535 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:58:37.919021 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916538 4852 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:58:37.919021 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916540 4852 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:58:37.919021 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916543 4852 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:58:37.919021 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916546 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:58:37.919021 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916549 4852 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:58:37.919021 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916551 4852 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:58:37.919021 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916554 4852 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:58:37.919485 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916557 4852 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:58:37.919485 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916576 4852 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:58:37.919485 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916579 4852 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:58:37.919485 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916582 4852 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:58:37.919485 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916585 4852 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:58:37.919485 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916587 4852 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:58:37.919485 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916590 4852 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:58:37.919485 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916592 4852 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:58:37.919485 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916595 4852 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:58:37.919485 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916598 4852 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:58:37.919485 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916600 4852 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:58:37.919485 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916603 4852 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:58:37.919485 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916605 4852 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:58:37.919485 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916608 4852 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:58:37.919485 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916610 4852 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:58:37.919485 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916613 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:58:37.919485 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916615 4852 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:58:37.919485 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916618 4852 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:58:37.919485 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916620 4852 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:58:37.919485 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916623 4852 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:58:37.919988 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916625 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:58:37.919988 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916627 4852 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:58:37.919988 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916629 4852 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:58:37.919988 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916632 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:58:37.919988 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916635 4852 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:58:37.919988 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916638 4852 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:58:37.919988 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916641 4852 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:58:37.919988 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916644 4852 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:58:37.919988 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916646 4852 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:58:37.919988 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916649 4852 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:58:37.919988 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916651 4852 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:58:37.919988 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916654 4852 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:58:37.919988 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916656 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:58:37.919988 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916659 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:58:37.919988 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916661 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:58:37.919988 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916663 4852 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:58:37.919988 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916666 4852 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:58:37.919988 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916668 4852 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:58:37.919988 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916671 4852 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:58:37.919988 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916673 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:58:37.920469 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916677 4852 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:58:37.920469 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916681 4852 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:58:37.920469 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916684 4852 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:58:37.920469 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916687 4852 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:58:37.920469 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916690 4852 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:58:37.920469 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916692 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:58:37.920469 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916695 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:58:37.920469 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916698 4852 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:58:37.920469 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916701 4852 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:58:37.920469 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916703 4852 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:58:37.920469 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916706 4852 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:58:37.920469 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916708 4852 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:58:37.920469 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916711 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:58:37.920469 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.916713 4852 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:58:37.920469 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917173 4852 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:58:37.920469 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917178 4852 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:58:37.920469 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917182 4852 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:58:37.920469 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917186 4852 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:58:37.920469 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917189 4852 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:58:37.920941 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917192 4852 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:58:37.920941 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917195 4852 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:58:37.920941 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917198 4852 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:58:37.920941 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917200 4852 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:58:37.920941 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917203 4852 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:58:37.920941 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917205 4852 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:58:37.920941 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917208 4852 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:58:37.920941 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917210 4852 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:58:37.920941 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917213 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:58:37.920941 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917216 4852 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:58:37.920941 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917218 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:58:37.920941 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917220 4852 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:58:37.920941 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917223 4852 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:58:37.920941 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917225 4852 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:58:37.920941 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917228 4852 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:58:37.920941 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917231 4852 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:58:37.920941 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917233 4852 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:58:37.920941 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917236 4852 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:58:37.920941 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917239 4852 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:58:37.920941 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917242 4852 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:58:37.921409 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917244 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:58:37.921409 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917247 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:58:37.921409 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917250 4852 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:58:37.921409 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917253 4852 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:58:37.921409 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917255 4852 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:58:37.921409 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917258 4852 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:58:37.921409 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917260 4852 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:58:37.921409 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917262 4852 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:58:37.921409 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917265 4852 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:58:37.921409 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917267 4852 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:58:37.921409 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917270 4852 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:58:37.921409 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917273 4852 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:58:37.921409 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917275 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:58:37.921409 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917278 4852 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:58:37.921409 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917281 4852 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:58:37.921409 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917283 4852 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:58:37.921409 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917285 4852 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:58:37.921409 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917288 4852 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:58:37.921409 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917290 4852 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:58:37.921409 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917293 4852 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:58:37.921905 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917295 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:58:37.921905 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917298 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:58:37.921905 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917300 4852 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:58:37.921905 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917302 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:58:37.921905 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917305 4852 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:58:37.921905 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917308 4852 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:58:37.921905 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917311 4852 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:58:37.921905 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917313 4852 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:58:37.921905 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917316 4852 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:58:37.921905 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917318 4852 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:58:37.921905 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917321 4852 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:58:37.921905 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917323 4852 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:58:37.921905 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917326 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:58:37.921905 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917328 4852 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:58:37.921905 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917331 4852 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:58:37.921905 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917333 4852 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:58:37.921905 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917336 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:58:37.921905 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917338 4852 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:58:37.921905 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917341 4852 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:58:37.921905 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917343 4852 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:58:37.922378 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917345 4852 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:58:37.922378 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917348 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:58:37.922378 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917350 4852 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:58:37.922378 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917353 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:58:37.922378 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917356 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:58:37.922378 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917359 4852 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:58:37.922378 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917361 4852 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:58:37.922378 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917363 4852 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:58:37.922378 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917366 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:58:37.922378 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917368 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:58:37.922378 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917371 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:58:37.922378 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917373 4852 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:58:37.922378 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917375 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:58:37.922378 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917378 4852 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:58:37.922378 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917380 4852 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:58:37.922378 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917383 4852 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:58:37.922378 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917385 4852 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:58:37.922378 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917388 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:58:37.922378 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917391 4852 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:58:37.922378 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917393 4852 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:58:37.922887 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917397 4852 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:58:37.922887 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917401 4852 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:58:37.922887 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917403 4852 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:58:37.922887 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917406 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:58:37.922887 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917408 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:58:37.922887 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917411 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:58:37.922887 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917415 4852 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:58:37.922887 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917419 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:58:37.922887 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917422 4852 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:58:37.922887 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917424 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:58:37.922887 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917427 4852 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:58:37.922887 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917430 4852 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:58:37.922887 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917433 4852 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:58:37.922887 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917436 4852 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:58:37.922887 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917438 4852 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:58:37.922887 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917443 4852 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:58:37.922887 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917446 4852 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:58:37.922887 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917448 4852 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917451 4852 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917453 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.917456 4852 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917549 4852 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917556 4852 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917580 4852 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917586 4852 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917593 4852 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917597 4852 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917601 4852 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917605 4852 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917608 4852 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917611 4852 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917614 4852 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917617 4852 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917620 4852 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917623 4852 flags.go:64] FLAG: --cgroup-root="" Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917626 4852 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917629 4852 flags.go:64] FLAG: --client-ca-file="" Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917632 4852 flags.go:64] FLAG: --cloud-provider="external" Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917634 4852 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917639 4852 flags.go:64] FLAG: --cluster-domain="" Nov 04 00:58:37.923308 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917642 4852 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917645 4852 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917648 4852 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917651 4852 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917655 4852 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917658 4852 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917661 4852 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917666 4852 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917669 4852 flags.go:64] FLAG: --contention-profiling="false" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917672 4852 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917675 4852 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917678 4852 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917681 4852 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917684 4852 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917687 4852 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917690 4852 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917693 4852 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917695 4852 flags.go:64] FLAG: --enable-server="true" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917698 4852 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917702 4852 flags.go:64] FLAG: --event-burst="100" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917705 4852 flags.go:64] FLAG: --event-qps="50" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917707 4852 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917710 4852 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917713 4852 flags.go:64] FLAG: --eviction-hard="" Nov 04 00:58:37.923861 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917717 4852 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917720 4852 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917723 4852 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917726 4852 flags.go:64] FLAG: --eviction-soft="" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917729 4852 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917732 4852 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917735 4852 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917737 4852 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917740 4852 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917745 4852 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917748 4852 flags.go:64] FLAG: --feature-gates="" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917752 4852 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917755 4852 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917758 4852 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917761 4852 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917765 4852 flags.go:64] FLAG: --healthz-port="10248" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917768 4852 flags.go:64] FLAG: --help="false" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917771 4852 flags.go:64] FLAG: --hostname-override="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917775 4852 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917778 4852 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917780 4852 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917784 4852 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917787 4852 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 00:58:37.924474 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917800 4852 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917803 4852 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917806 4852 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917809 4852 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917812 4852 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917815 4852 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917818 4852 flags.go:64] FLAG: --kube-reserved="" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917820 4852 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917823 4852 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917826 4852 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917829 4852 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917831 4852 flags.go:64] FLAG: --lock-file="" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917834 4852 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917837 4852 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917840 4852 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917845 4852 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917848 4852 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917851 4852 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917856 4852 flags.go:64] FLAG: --logging-format="text" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917858 4852 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917862 4852 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917864 4852 flags.go:64] FLAG: --manifest-url="" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917867 4852 flags.go:64] FLAG: --manifest-url-header="" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917871 4852 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917876 4852 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 00:58:37.925033 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917880 4852 flags.go:64] FLAG: --max-pods="110" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917882 4852 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917885 4852 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917889 4852 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917892 4852 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917895 4852 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917898 4852 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917900 4852 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917907 4852 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917910 4852 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917913 4852 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917916 4852 flags.go:64] FLAG: --pod-cidr="" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917918 4852 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917921 4852 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917924 4852 flags.go:64] FLAG: --pods-per-core="0" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917927 4852 flags.go:64] FLAG: --port="10250" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917929 4852 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917932 4852 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0312e0f14a1f7850b" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917935 4852 flags.go:64] FLAG: --qos-reserved="" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917938 4852 flags.go:64] FLAG: --read-only-port="10255" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917941 4852 flags.go:64] FLAG: --register-node="true" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917944 4852 flags.go:64] FLAG: --register-with-taints="" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917947 4852 flags.go:64] FLAG: --registry-burst="10" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917950 4852 flags.go:64] FLAG: --registry-qps="5" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917952 4852 flags.go:64] FLAG: --reserved-cpus="" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917956 4852 flags.go:64] FLAG: --reserved-memory="" Nov 04 00:58:37.925659 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917960 4852 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917963 4852 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917965 4852 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917968 4852 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917971 4852 flags.go:64] FLAG: --runonce="false" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917976 4852 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917979 4852 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917982 4852 flags.go:64] FLAG: --seccomp-default="false" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917984 4852 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917987 4852 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917991 4852 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917994 4852 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917997 4852 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.917999 4852 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.918002 4852 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.918004 4852 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.918007 4852 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.918010 4852 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.918013 4852 flags.go:64] FLAG: --system-cgroups="" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.918016 4852 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.918021 4852 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.918023 4852 flags.go:64] FLAG: --tls-cert-file="" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.918026 4852 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.918036 4852 flags.go:64] FLAG: --tls-min-version="" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.918039 4852 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 00:58:37.926281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.918042 4852 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 00:58:37.926893 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.918045 4852 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 00:58:37.926893 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.918048 4852 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 00:58:37.926893 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.918050 4852 flags.go:64] FLAG: --v="2" Nov 04 00:58:37.926893 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.918059 4852 flags.go:64] FLAG: --version="false" Nov 04 00:58:37.926893 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.918063 4852 flags.go:64] FLAG: --vmodule="" Nov 04 00:58:37.926893 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.918068 4852 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 00:58:37.926893 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.918071 4852 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 00:58:37.926893 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918218 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:58:37.926893 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918222 4852 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:58:37.926893 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918225 4852 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:58:37.926893 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918227 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:58:37.926893 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918231 4852 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:58:37.926893 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918234 4852 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:58:37.926893 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918236 4852 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:58:37.926893 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918239 4852 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:58:37.926893 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918242 4852 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:58:37.926893 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918244 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:58:37.926893 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918247 4852 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:58:37.926893 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918250 4852 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:58:37.926893 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918252 4852 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:58:37.926893 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918255 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:58:37.927516 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918257 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:58:37.927516 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918260 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:58:37.927516 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918262 4852 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:58:37.927516 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918265 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:58:37.927516 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918267 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:58:37.927516 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918270 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:58:37.927516 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918272 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:58:37.927516 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918275 4852 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:58:37.927516 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918277 4852 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:58:37.927516 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918280 4852 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:58:37.927516 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918282 4852 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:58:37.927516 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918284 4852 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:58:37.927516 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918287 4852 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:58:37.927516 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918289 4852 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:58:37.927516 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918292 4852 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:58:37.927516 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918295 4852 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:58:37.927516 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918299 4852 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:58:37.927516 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918301 4852 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:58:37.927516 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918304 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:58:37.928272 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918307 4852 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:58:37.928272 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918309 4852 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:58:37.928272 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918312 4852 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:58:37.928272 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918316 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:58:37.928272 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918318 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:58:37.928272 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918321 4852 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:58:37.928272 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918323 4852 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:58:37.928272 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918325 4852 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:58:37.928272 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918328 4852 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:58:37.928272 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918330 4852 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:58:37.928272 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918333 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:58:37.928272 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918336 4852 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:58:37.928272 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918338 4852 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:58:37.928272 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918340 4852 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:58:37.928272 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918343 4852 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:58:37.928272 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918346 4852 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:58:37.928272 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918348 4852 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:58:37.928272 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918351 4852 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:58:37.928272 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918353 4852 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:58:37.928272 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918356 4852 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:58:37.929104 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918358 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:58:37.929104 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918361 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:58:37.929104 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918363 4852 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:58:37.929104 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918366 4852 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:58:37.929104 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918368 4852 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:58:37.929104 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918370 4852 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:58:37.929104 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918373 4852 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:58:37.929104 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918375 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:58:37.929104 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918378 4852 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:58:37.929104 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918383 4852 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:58:37.929104 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918386 4852 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:58:37.929104 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918389 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:58:37.929104 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918392 4852 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:58:37.929104 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918395 4852 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:58:37.929104 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918397 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:58:37.929104 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918402 4852 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:58:37.929104 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918404 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:58:37.929104 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918417 4852 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:58:37.929104 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918420 4852 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:58:37.929853 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918423 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:58:37.929853 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918425 4852 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:58:37.929853 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918428 4852 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:58:37.929853 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918430 4852 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:58:37.929853 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918433 4852 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:58:37.929853 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918436 4852 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:58:37.929853 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918439 4852 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:58:37.929853 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918441 4852 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:58:37.929853 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918443 4852 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:58:37.929853 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918446 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:58:37.929853 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918448 4852 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:58:37.929853 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918451 4852 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:58:37.929853 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918453 4852 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:58:37.929853 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918456 4852 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:58:37.929853 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918458 4852 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:58:37.929853 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918461 4852 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:58:37.929853 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918463 4852 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:58:37.929853 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918466 4852 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:58:37.929853 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918468 4852 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:58:37.929853 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918471 4852 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:58:37.930518 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918473 4852 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:58:37.930518 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918476 4852 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:58:37.930518 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918480 4852 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:58:37.930518 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918482 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:58:37.930518 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918485 4852 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:58:37.930518 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918487 4852 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:58:37.930518 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918490 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:58:37.930518 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918492 4852 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:58:37.930518 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918496 4852 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:58:37.930518 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918499 4852 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:58:37.930518 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918502 4852 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:58:37.930518 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918504 4852 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:58:37.930518 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918507 4852 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:58:37.930518 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918511 4852 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:58:37.930518 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.918536 4852 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:58:37.930518 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918589 4852 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:58:37.931196 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918593 4852 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:58:37.931196 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918597 4852 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:58:37.931196 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918600 4852 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:58:37.931196 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918603 4852 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:58:37.931196 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918606 4852 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:58:37.931196 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918609 4852 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:58:37.931196 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918612 4852 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:58:37.931196 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918615 4852 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:58:37.931196 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918618 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:58:37.931196 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918620 4852 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:58:37.931196 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918623 4852 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:58:37.931196 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918625 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:58:37.931196 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918628 4852 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:58:37.931196 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918631 4852 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:58:37.931196 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918633 4852 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:58:37.931196 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918636 4852 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:58:37.931196 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918638 4852 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:58:37.931196 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918640 4852 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:58:37.931196 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918644 4852 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:58:37.931196 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918649 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:58:37.931747 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918652 4852 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:58:37.931747 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918656 4852 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:58:37.931747 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918658 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:58:37.931747 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918661 4852 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:58:37.931747 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918664 4852 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:58:37.931747 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918667 4852 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:58:37.931747 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918669 4852 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:58:37.931747 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918672 4852 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:58:37.931747 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918675 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:58:37.931747 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918677 4852 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:58:37.931747 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918679 4852 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:58:37.931747 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918682 4852 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:58:37.931747 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918684 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:58:37.931747 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918687 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:58:37.931747 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918691 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:58:37.931747 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918693 4852 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:58:37.931747 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918696 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:58:37.931747 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918698 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:58:37.931747 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918701 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:58:37.932195 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918704 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:58:37.932195 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918706 4852 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:58:37.932195 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918709 4852 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:58:37.932195 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918711 4852 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:58:37.932195 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918714 4852 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:58:37.932195 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918716 4852 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:58:37.932195 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918718 4852 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:58:37.932195 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918721 4852 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:58:37.932195 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918723 4852 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:58:37.932195 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918726 4852 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:58:37.932195 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918729 4852 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:58:37.932195 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918731 4852 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:58:37.932195 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918735 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:58:37.932195 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918737 4852 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:58:37.932195 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918740 4852 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:58:37.932195 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918742 4852 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:58:37.932195 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918745 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:58:37.932195 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918747 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:58:37.932195 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918750 4852 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:58:37.932967 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918752 4852 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:58:37.932967 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918755 4852 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:58:37.932967 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918757 4852 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:58:37.932967 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918760 4852 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:58:37.932967 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918762 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:58:37.932967 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918777 4852 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:58:37.932967 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918780 4852 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:58:37.932967 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918784 4852 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:58:37.932967 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918787 4852 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:58:37.932967 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918790 4852 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:58:37.932967 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918793 4852 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:58:37.932967 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918796 4852 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:58:37.932967 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918798 4852 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:58:37.932967 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918801 4852 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:58:37.932967 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918803 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:58:37.932967 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918806 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:58:37.932967 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918808 4852 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:58:37.932967 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918811 4852 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:58:37.932967 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918813 4852 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:58:37.932967 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918816 4852 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:58:37.933522 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918819 4852 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:58:37.933522 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918821 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:58:37.933522 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918823 4852 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:58:37.933522 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918827 4852 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:58:37.933522 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918831 4852 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:58:37.933522 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918835 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:58:37.933522 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918837 4852 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:58:37.933522 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918840 4852 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:58:37.933522 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918842 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:58:37.933522 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918844 4852 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:58:37.933522 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918847 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:58:37.933522 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918849 4852 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:58:37.933522 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918855 4852 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:58:37.933522 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918858 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:58:37.933522 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918860 4852 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:58:37.933522 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918863 4852 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:58:37.933522 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918865 4852 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:58:37.933522 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918868 4852 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:58:37.933522 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918870 4852 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:58:37.934016 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918872 4852 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:58:37.934016 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918875 4852 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:58:37.934016 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918877 4852 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:58:37.934016 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918880 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:58:37.934016 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918883 4852 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:58:37.934016 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918885 4852 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:58:37.934016 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918888 4852 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:58:37.934016 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.918890 4852 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:58:37.934016 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.922804 4852 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 00:58:37.934016 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.922818 4852 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 00:58:37.934016 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.922834 4852 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 00:58:37.934016 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.922839 4852 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 00:58:37.934016 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922909 4852 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:58:37.934016 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922914 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:58:37.934016 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922917 4852 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:58:37.934016 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922921 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:58:37.934016 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922923 4852 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:58:37.934016 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922926 4852 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:58:37.934016 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922929 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:58:37.934482 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922931 4852 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:58:37.934482 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922934 4852 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:58:37.934482 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922937 4852 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:58:37.934482 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922939 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:58:37.934482 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922942 4852 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:58:37.934482 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922945 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:58:37.934482 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922948 4852 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:58:37.934482 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922950 4852 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:58:37.934482 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922953 4852 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:58:37.934482 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922955 4852 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:58:37.934482 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922958 4852 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:58:37.934482 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922961 4852 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:58:37.934482 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922964 4852 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:58:37.934482 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922967 4852 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:58:37.934482 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922970 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:58:37.934482 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922972 4852 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:58:37.934482 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922975 4852 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:58:37.934482 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922978 4852 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:58:37.934482 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922980 4852 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:58:37.934959 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922983 4852 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:58:37.934959 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922986 4852 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:58:37.934959 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922988 4852 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:58:37.934959 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922990 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:58:37.934959 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922993 4852 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:58:37.934959 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922995 4852 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:58:37.934959 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.922998 4852 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:58:37.934959 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923000 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:58:37.934959 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923003 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:58:37.934959 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923005 4852 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:58:37.934959 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923008 4852 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:58:37.934959 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923010 4852 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:58:37.934959 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923013 4852 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:58:37.934959 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923015 4852 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:58:37.934959 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923017 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:58:37.934959 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923020 4852 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:58:37.934959 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923022 4852 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:58:37.934959 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923025 4852 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:58:37.934959 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923027 4852 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:58:37.934959 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923030 4852 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:58:37.935511 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923033 4852 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:58:37.935511 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923035 4852 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:58:37.935511 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923038 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:58:37.935511 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923040 4852 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:58:37.935511 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923043 4852 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:58:37.935511 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923045 4852 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:58:37.935511 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923048 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:58:37.935511 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923050 4852 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:58:37.935511 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923053 4852 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:58:37.935511 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923055 4852 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:58:37.935511 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923058 4852 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:58:37.935511 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923061 4852 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:58:37.935511 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923063 4852 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:58:37.935511 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923066 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:58:37.935511 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923068 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:58:37.935511 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923071 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:58:37.935511 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923073 4852 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:58:37.935511 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923075 4852 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:58:37.935511 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923078 4852 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:58:37.935511 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923080 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:58:37.936070 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923082 4852 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:58:37.936070 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923085 4852 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:58:37.936070 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923087 4852 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:58:37.936070 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923089 4852 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:58:37.936070 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923092 4852 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:58:37.936070 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923095 4852 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:58:37.936070 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923099 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:58:37.936070 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923102 4852 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:58:37.936070 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923105 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:58:37.936070 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923107 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:58:37.936070 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923109 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:58:37.936070 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923112 4852 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:58:37.936070 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923114 4852 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:58:37.936070 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923116 4852 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:58:37.936070 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923119 4852 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:58:37.936070 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923121 4852 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:58:37.936070 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923124 4852 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:58:37.936070 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923127 4852 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:58:37.936070 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923129 4852 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:58:37.936528 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923132 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:58:37.936528 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923134 4852 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:58:37.936528 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923136 4852 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:58:37.936528 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923139 4852 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:58:37.936528 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923144 4852 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:58:37.936528 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923147 4852 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:58:37.936528 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923149 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:58:37.936528 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923153 4852 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:58:37.936528 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923157 4852 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:58:37.936528 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923159 4852 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:58:37.936528 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923162 4852 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:58:37.936528 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923164 4852 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:58:37.936528 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923167 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:58:37.936528 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923169 4852 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:58:37.936528 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923172 4852 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:58:37.936528 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923174 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:58:37.936528 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923177 4852 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:58:37.936528 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923179 4852 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:58:37.936528 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923181 4852 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:58:37.937077 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923183 4852 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:58:37.937077 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923186 4852 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:58:37.937077 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.923212 4852 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:58:37.937077 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923388 4852 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:58:37.937077 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923393 4852 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:58:37.937077 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923395 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:58:37.937077 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923398 4852 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:58:37.937077 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923400 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:58:37.937077 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923403 4852 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:58:37.937077 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923405 4852 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:58:37.937077 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923408 4852 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:58:37.937077 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923411 4852 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:58:37.937077 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923414 4852 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:58:37.937077 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923416 4852 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:58:37.937077 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923418 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:58:37.937077 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923421 4852 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:58:37.937464 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923423 4852 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:58:37.937464 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923426 4852 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:58:37.937464 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923433 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:58:37.937464 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923436 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:58:37.937464 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923439 4852 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:58:37.937464 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923441 4852 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:58:37.937464 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923444 4852 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:58:37.937464 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923446 4852 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:58:37.937464 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923448 4852 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:58:37.937464 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923451 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:58:37.937464 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923453 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:58:37.937464 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923455 4852 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:58:37.937464 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923458 4852 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:58:37.937464 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923460 4852 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:58:37.937464 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923463 4852 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:58:37.937464 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923465 4852 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:58:37.937464 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923467 4852 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:58:37.937464 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923471 4852 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:58:37.937464 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923475 4852 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:58:37.937970 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923477 4852 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:58:37.937970 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923480 4852 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:58:37.937970 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923483 4852 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:58:37.937970 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923485 4852 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:58:37.937970 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923487 4852 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:58:37.937970 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923490 4852 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:58:37.937970 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923492 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:58:37.937970 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923495 4852 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:58:37.937970 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923497 4852 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:58:37.937970 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923499 4852 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:58:37.937970 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923502 4852 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:58:37.937970 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923504 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:58:37.937970 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923507 4852 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:58:37.937970 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923509 4852 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:58:37.937970 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923511 4852 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:58:37.937970 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923514 4852 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:58:37.937970 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923522 4852 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:58:37.937970 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923525 4852 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:58:37.937970 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923527 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:58:37.937970 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923529 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:58:37.938712 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923532 4852 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:58:37.938712 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923534 4852 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:58:37.938712 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923538 4852 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:58:37.938712 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923541 4852 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:58:37.938712 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923544 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:58:37.938712 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923547 4852 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:58:37.938712 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923549 4852 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:58:37.938712 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923552 4852 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:58:37.938712 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923554 4852 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:58:37.938712 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923556 4852 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:58:37.938712 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923573 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:58:37.938712 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923576 4852 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:58:37.938712 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923578 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:58:37.938712 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923581 4852 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:58:37.938712 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923583 4852 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:58:37.938712 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923586 4852 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:58:37.938712 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923588 4852 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:58:37.938712 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923590 4852 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:58:37.938712 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923593 4852 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:58:37.939301 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923595 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:58:37.939301 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923598 4852 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:58:37.939301 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923600 4852 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:58:37.939301 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923603 4852 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:58:37.939301 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923605 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:58:37.939301 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923608 4852 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:58:37.939301 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923610 4852 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:58:37.939301 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923612 4852 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:58:37.939301 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923615 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:58:37.939301 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923617 4852 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:58:37.939301 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923625 4852 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:58:37.939301 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923628 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:58:37.939301 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923630 4852 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:58:37.939301 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923633 4852 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:58:37.939301 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923635 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:58:37.939301 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923637 4852 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:58:37.939301 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923640 4852 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:58:37.939301 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923642 4852 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:58:37.939301 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923644 4852 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:58:37.939301 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923647 4852 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:58:37.940041 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923650 4852 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:58:37.940041 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923652 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:58:37.940041 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923654 4852 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:58:37.940041 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923657 4852 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:58:37.940041 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923659 4852 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:58:37.940041 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923662 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:58:37.940041 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923664 4852 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:58:37.940041 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923667 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:58:37.940041 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923669 4852 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:58:37.940041 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923671 4852 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:58:37.940041 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923674 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:58:37.940041 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923676 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:58:37.940041 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923679 4852 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:58:37.940041 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923681 4852 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:58:37.940041 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923683 4852 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:58:37.940041 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.923707 4852 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:58:37.940627 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923752 4852 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:58:37.940627 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923757 4852 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:58:37.940627 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923760 4852 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:58:37.940627 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923762 4852 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:58:37.940627 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923765 4852 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:58:37.940627 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923769 4852 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:58:37.940627 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923773 4852 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:58:37.940627 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923782 4852 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:58:37.940627 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923785 4852 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:58:37.940627 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923788 4852 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:58:37.940627 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923791 4852 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:58:37.940627 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923793 4852 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:58:37.940627 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923796 4852 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:58:37.940627 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923798 4852 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:58:37.940627 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923800 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:58:37.940627 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923803 4852 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:58:37.940627 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923805 4852 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:58:37.940627 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923808 4852 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:58:37.940627 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923810 4852 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:58:37.941172 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923813 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:58:37.941172 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923815 4852 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:58:37.941172 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923818 4852 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:58:37.941172 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923820 4852 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:58:37.941172 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923823 4852 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:58:37.941172 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923825 4852 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:58:37.941172 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923827 4852 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:58:37.941172 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923830 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:58:37.941172 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923832 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:58:37.941172 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923835 4852 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:58:37.941172 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923838 4852 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:58:37.941172 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923842 4852 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:58:37.941172 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923851 4852 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:58:37.941172 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923854 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:58:37.941172 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923856 4852 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:58:37.941172 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923859 4852 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:58:37.941172 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923861 4852 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:58:37.941172 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923864 4852 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:58:37.941172 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923868 4852 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:58:37.941637 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923870 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:58:37.941637 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923873 4852 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:58:37.941637 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923881 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:58:37.941637 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923884 4852 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:58:37.941637 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923886 4852 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:58:37.941637 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923889 4852 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:58:37.941637 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923891 4852 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:58:37.941637 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923894 4852 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:58:37.941637 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923896 4852 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:58:37.941637 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923898 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:58:37.941637 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923901 4852 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:58:37.941637 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923903 4852 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:58:37.941637 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923906 4852 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:58:37.941637 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923908 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:58:37.941637 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923911 4852 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:58:37.941637 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923913 4852 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:58:37.941637 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923916 4852 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:58:37.941637 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923918 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:58:37.941637 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923920 4852 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:58:37.942124 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923923 4852 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:58:37.942124 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923925 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:58:37.942124 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923927 4852 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:58:37.942124 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923931 4852 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:58:37.942124 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923934 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:58:37.942124 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923936 4852 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:58:37.942124 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923939 4852 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:58:37.942124 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923941 4852 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:58:37.942124 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923943 4852 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:58:37.942124 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923946 4852 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:58:37.942124 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923948 4852 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:58:37.942124 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923951 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:58:37.942124 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923956 4852 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:58:37.942124 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923958 4852 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:58:37.942124 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923961 4852 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:58:37.942124 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923963 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:58:37.942124 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923971 4852 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:58:37.942124 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923974 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:58:37.942124 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923976 4852 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:58:37.942124 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923979 4852 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:58:37.942670 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923981 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:58:37.942670 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923984 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:58:37.942670 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923986 4852 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:58:37.942670 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923989 4852 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:58:37.942670 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923991 4852 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:58:37.942670 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923994 4852 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:58:37.942670 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923996 4852 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:58:37.942670 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.923999 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:58:37.942670 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.924001 4852 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:58:37.942670 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.924003 4852 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:58:37.942670 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.924006 4852 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:58:37.942670 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.924008 4852 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:58:37.942670 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.924010 4852 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:58:37.942670 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.924013 4852 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:58:37.942670 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.924015 4852 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:58:37.942670 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.924019 4852 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:58:37.942670 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.924022 4852 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:58:37.942670 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.924024 4852 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:58:37.942670 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.924027 4852 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:58:37.943153 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.924030 4852 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:58:37.943153 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.924032 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:58:37.943153 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.924034 4852 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:58:37.943153 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.924037 4852 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:58:37.943153 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.924039 4852 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:58:37.943153 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.924043 4852 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:58:37.943153 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.924045 4852 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:58:37.943153 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.924052 4852 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:58:37.943153 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.924055 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:58:37.943153 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:37.924057 4852 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:58:37.943153 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.924110 4852 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 00:58:37.943153 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.925106 4852 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 04 00:58:37.943153 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.925165 4852 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 04 00:58:37.943153 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.925365 4852 server.go:1014] "Starting client certificate rotation" Nov 04 00:58:37.943153 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.925507 4852 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 00:58:37.943153 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.925554 4852 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 00:52:55 +0000 UTC" deadline="2028-06-09 15:14:42.069767764 +0000 UTC" Nov 04 00:58:37.943807 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.925591 4852 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14006h16m4.144178421s" Nov 04 00:58:37.943807 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.925923 4852 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 00:58:37.943807 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.927100 4852 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 00:58:37.943807 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.930615 4852 log.go:25] "Validated CRI v1 runtime API" Nov 04 00:58:37.943807 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.932344 4852 log.go:25] "Validated CRI v1 image API" Nov 04 00:58:37.943807 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.933179 4852 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 00:58:37.943807 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.934837 4852 fs.go:135] Filesystem UUIDs: map[04f6d7dc-ee37-48f4-97f3-b2eb09cef2f0:/dev/nvme0n1p3 709d32f6-32ce-45ff-ad33-917e30cecb0c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 04 00:58:37.944003 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.934867 4852 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/116018e6f7a57844a290bbeafac250cf3eec2bd16bd247250c1589e0ee539b7c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/116018e6f7a57844a290bbeafac250cf3eec2bd16bd247250c1589e0ee539b7c/userdata/shm major:0 minor:103 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b0a4b0b861d58a2c7583fedabd3e2e24771e738f0e5ece34c94211c52375518/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b0a4b0b861d58a2c7583fedabd3e2e24771e738f0e5ece34c94211c52375518/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/277ddfb0de2eee6d0a584457c024d05cb0c5fa9ced843e754720beae14816406/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/277ddfb0de2eee6d0a584457c024d05cb0c5fa9ced843e754720beae14816406/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c2ff0ae8b6107ced8dc11426662d452592f20733f317e1f9d573ff3f9f48ddb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c2ff0ae8b6107ced8dc11426662d452592f20733f317e1f9d573ff3f9f48ddb/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3342c8d9052dfe9f3803d0db938c962dc58657ee79a9802b49e5be497a29772b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3342c8d9052dfe9f3803d0db938c962dc58657ee79a9802b49e5be497a29772b/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3fc4c1e4b8b2fb772b8257e1316188ed74a737fd023c3acf4d90b68c20be30c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3fc4c1e4b8b2fb772b8257e1316188ed74a737fd023c3acf4d90b68c20be30c8/userdata/shm major:0 minor:267 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4832f0e32b037011ba261e4a227f95b569e553145a1aec19d3bd81bebceef542/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4832f0e32b037011ba261e4a227f95b569e553145a1aec19d3bd81bebceef542/userdata/shm major:0 minor:108 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/652f3da9f495dbf0ada3f6694492cdcaf14feec9c4f9038817871fec701a498a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/652f3da9f495dbf0ada3f6694492cdcaf14feec9c4f9038817871fec701a498a/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c348450f2fa7f6112786c73979ebec84d5fbc267418d8b91e15825916c9eefd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c348450f2fa7f6112786c73979ebec84d5fbc267418d8b91e15825916c9eefd/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/928b0878ac73f0c4f75f2d9253aa831dae7a8d135a26d976dbd74a8cf4f1f552/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/928b0878ac73f0c4f75f2d9253aa831dae7a8d135a26d976dbd74a8cf4f1f552/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95ffdb116aae4604d44141563ca1e3486d047bf765addbbb136234608ae2b4c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95ffdb116aae4604d44141563ca1e3486d047bf765addbbb136234608ae2b4c4/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97ca9fe8c890a540b9403fd94faf0edb071532c8f5e838a5a6b274bb9ed59f51/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97ca9fe8c890a540b9403fd94faf0edb071532c8f5e838a5a6b274bb9ed59f51/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b0240716b1410bacd943746df2af03430f99980a3e0acc79aa62c401898a250/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b0240716b1410bacd943746df2af03430f99980a3e0acc79aa62c401898a250/userdata/shm major:0 minor:260 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b097049413e19ac6bbccd2cd05485546e0b37baf228f313ac7f97a283096122/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b097049413e19ac6bbccd2cd05485546e0b37baf228f313ac7f97a283096122/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac7c69deda762ae8537940d4a40a5047556167936c3d3932d3fc617ce507b23d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac7c69deda762ae8537940d4a40a5047556167936c3d3932d3fc617ce507b23d/userdata/shm major:0 minor:100 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af1da2809571589a0f36763d612938735298294e863cbf70f0576500fab216b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af1da2809571589a0f36763d612938735298294e863cbf70f0576500fab216b5/userdata/shm major:0 minor:120 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5c23d2cad0687c62de401a72caf2927884cc276b0be1f6d294366179128eaa7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5c23d2cad0687c62de401a72caf2927884cc276b0be1f6d294366179128eaa7/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec/volumes/kubernetes.io~projected/kube-api-access-llvc6:{mountpoint:/var/lib/kubelet/pods/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec/volumes/kubernetes.io~projected/kube-api-access-llvc6 major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c/volumes/kubernetes.io~projected/kube-api-access-drqhw:{mountpoint:/var/lib/kubelet/pods/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c/volumes/kubernetes.io~projected/kube-api-access-drqhw major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c/volumes/kubernetes.io~secret/metrics-tls major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/402ccc5f-d755-4061-a96b-8ddc0495b711/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/402ccc5f-d755-4061-a96b-8ddc0495b711/volumes/kubernetes.io~secret/agent-certs major:0 minor:71 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47fb874a-90bd-41f9-9a8c-0e920d789ca2/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/47fb874a-90bd-41f9-9a8c-0e920d789ca2/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55e1fd53-92c2-484e-9405-ac5af81ed61b/volumes/kubernetes.io~projected/kube-api-access-m8q28:{mountpoint:/var/lib/kubelet/pods/55e1fd53-92c2-484e-9405-ac5af81ed61b/volumes/kubernetes.io~projected/kube-api-access-m8q28 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55e1fd53-92c2-484e-9405-ac5af81ed61b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/55e1fd53-92c2-484e-9405-ac5af81ed61b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/565a71a6-1ce7-4078-90fc-f2ccadaa0d17/volumes/kubernetes.io~projected/kube-api-access-npmf5:{mountpoint:/var/lib/kubelet/pods/565a71a6-1ce7-4078-90fc-f2ccadaa0d17/volumes/kubernetes.io~projected/kube-api-access-npmf5 major:0 minor:74 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d50c29d-9104-4dd5-9499-7dd94594666b/volumes/kubernetes.io~projected/kube-api-access-65cwq:{mountpoint:/var/lib/kubelet/pods/5d50c29d-9104-4dd5-9499-7dd94594666b/volumes/kubernetes.io~projected/kube-api-access-65cwq major:0 minor:114 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d50c29d-9104-4dd5-9499-7dd94594666b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5d50c29d-9104-4dd5-9499-7dd94594666b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:119 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d50c29d-9104-4dd5-9499-7dd94594666b/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/5d50c29d-9104-4dd5-9499-7dd94594666b/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:118 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66117eac-ae66-462e-8282-7ef04b047fbe/volumes/kubernetes.io~projected/kube-api-access-8vt9h:{mountpoint:/var/lib/kubelet/pods/66117eac-ae66-462e-8282-7ef04b047fbe/volumes/kubernetes.io~projected/kube-api-access-8vt9h major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66117eac-ae66-462e-8282-7ef04b047fbe/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/66117eac-ae66-462e-8282-7ef04b047fbe/volumes/kubernetes.io~secret/cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cbe8a51-0318-450b-adb2-4abfe1097883/volumes/kubernetes.io~projected/kube-api-access-xttk5:{mountpoint:/var/lib/kubelet/pods/7cbe8a51-0318-450b-adb2-4abfe1097883/volumes/kubernetes.io~projected/kube-api-access-xttk5 major:0 minor:68 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/847ad3d1-d945-4281-9a5b-15420289c70d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/847ad3d1-d945-4281-9a5b-15420289c70d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/847ad3d1-d945-4281-9a5b-15420289c70d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/847ad3d1-d945-4281-9a5b-15420289c70d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/847ad3d1-d945-4281-9a5b-15420289c70d/volumes/kubernetes.io~projected/kube-api-access-j7s8t:{mountpoint:/var/lib/kubelet/pods/847ad3d1-d945-4281-9a5b-15420289c70d/volumes/kubernetes.io~projected/kube-api-access-j7s8t major:0 minor:70 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/975903c9-9da2-4971-a50b-7bcc37df6b34/volumes/kubernetes.io~projected/kube-api-access-bdntp:{mountpoint:/var/lib/kubelet/pods/975903c9-9da2-4971-a50b-7bcc37df6b34/volumes/kubernetes.io~projected/kube-api-access-bdntp major:0 minor:73 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac5e9e6b-7405-4c03-a149-4bf353209f24/volumes/kubernetes.io~projected/kube-api-access-xzwsb:{mountpoint:/var/lib/kubelet/pods/ac5e9e6b-7405-4c03-a149-4bf353209f24/volumes/kubernetes.io~projected/kube-api-access-xzwsb major:0 minor:72 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb2c2fb8-f28c-430b-8210-db7032a7073b/volumes/kubernetes.io~projected/kube-api-access-pdklr:{mountpoint:/var/lib/kubelet/pods/bb2c2fb8-f28c-430b-8210-db7032a7073b/volumes/kubernetes.io~projected/kube-api-access-pdklr major:0 minor:76 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb2c2fb8-f28c-430b-8210-db7032a7073b/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/bb2c2fb8-f28c-430b-8210-db7032a7073b/volumes/kubernetes.io~secret/metrics-certs major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6eada63-a80f-4eff-8076-6f01b9efa23e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c6eada63-a80f-4eff-8076-6f01b9efa23e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6eada63-a80f-4eff-8076-6f01b9efa23e/volumes/kubernetes.io~projected/kube-api-access-j8dw2:{mountpoint:/var/lib/kubelet/pods/c6eada63-a80f-4eff-8076-6f01b9efa23e/volumes/kubernetes.io~projected/kube-api-access-j8dw2 major:0 minor:69 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6eada63-a80f-4eff-8076-6f01b9efa23e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c6eada63-a80f-4eff-8076-6f01b9efa23e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6083261-f4cb-41ee-999d-8c5406473ff8/volumes/kubernetes.io~projected/kube-api-access-z8ms4:{mountpoint:/var/lib/kubelet/pods/f6083261-f4cb-41ee-999d-8c5406473ff8/volumes/kubernetes.io~projected/kube-api-access-z8ms4 major:0 minor:75 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f784deee-bbd7-4e50-94d1-7bb64fc61b91/volumes/kubernetes.io~projected/kube-api-access-htztj:{mountpoint:/var/lib/kubelet/pods/f784deee-bbd7-4e50-94d1-7bb64fc61b91/volumes/kubernetes.io~projected/kube-api-access-htztj major:0 minor:77 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/946a2473dbac981b75e5d55da8843bc16abd170a4ad014923fc0bc97af48ccd6/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/bbf0290c4c1f77bc1c6bcf6a254b4d7e9aab43f6fe30ab8fe97b21e6316b270d/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-110:{mountpoint:/var/lib/containers/storage/overlay/551ef4138d27bbf390eadb25e7d71f8f3bf603f23de9b94eacc73fd1352fc377/merged major:0 minor:110 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/2995146e94a9d4d2c8d818b31aae30897e5ad7e89b2f3271c0f0af9fba554a7e/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/0ca0274928c414d34e67cdc1292426579c5fc5c108e48683ceae1cd6fb45bb02/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-126:{mountpoint:/var/lib/containers/storage/overlay/619d09613008c65a0046547c85db011c279569cdc5c9ae52df2880acb190ff59/merged major:0 minor:126 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/c129ab26c42cff11a590966269d5134b6dcd376397988c101a72fc5fff4b3313/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/55e032069c224fa403aa59357313665979d5e3057a5092dfbfe0f15c1299b1d6/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/6b03cc4fd4a986177cee852b1c79f9b6ee842b64a41cd4f4e3ebac17faeb618f/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/23f9c5a96d541d2fbf32adee99018be81b3af25bacaaac5cf6ad079f891916cc/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/09f004bb576a2e9ca4b1977427700f4433357136c3c02c4e9be466587e6c120d/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/a6e67835f74c900cf24661f5666226ecdc2ee0c479b5c22fb72a5d725a4039df/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-156:{mountpoint:/var/lib/containers/storage/overlay/9803c9aaccb48cd0886dc05b79ef535a9388efc5dd9d78ebb91f014e85ec5ec8/merged major:0 minor:156 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/cbe3d4371d23252eb61dd6d1806aaa0adfd2c4e5d397625a08dc52eb26c8c6d5/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/02556f505ecb368408d23d460d0ba06a5e35f107a2a763b5c1926f44cfaa10fa/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/7ff4b0a990eb8eb4a3d8a71a402e4a06e184ba6102de8f8c030a1f04b001ebb8/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/var/lib/containers/storage/overlay/ae196b3dcbd2326cc1d14811b0137a30c2e10621bc90d8aed86860f7810324e4/merged major:0 minor:176 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/fe42c34fe266203ca58674feb33fc47072b426418c403b958811d94bf6dfa0ed/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-189:{mountpoint:/var/lib/containers/storage/overlay/2780c6fc6c6b439cbb989947fbbdb733069ffc24dc7cb3ca208979f6d8794799/merged major:0 minor:189 fsType:overlay blockSize:0} overlay_0-194:{mountpoint:/var/lib/containers/storage/overlay/50ba5f6901699ead2552abf1bb5400a338480ab739fee407ca517ad51a7c5ae3/merged major:0 minor:194 fsType:overlay blockSize:0} overlay_0-199:{mountpoint:/var/lib/containers/storage/overlay/35aff13edad0ac7faaa084e5a5705816b41306d0f6c5c9cece2058129aa467a6/merged major:0 minor:199 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/68200a90b13601abf4a69ef91dc08d9180ddb2412985feb19e74f03ae5f8197e/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/bfe0e989f816fdfe0021af57df675d31f1a49bde9bf5f4cecced2ce48076caa8/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/07391973ae7b0c941942da8c70b32f967a7217fb71544762089056daa9e7adfd/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/c2121f94abb0a1b6731739275702a1b0ee6e35ef8a07fef2b6f290c04fd1b588/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/362643031d3d776a018a7bb8d89029e006515e3778528652ef139ed82b9bf368/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/4f9225555cb0ad94e0b4e7b1e7b2269dd48fd2c94524322153273b2b6a057d56/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/de81234298afa182d352f20b6dcb792a7f868a9eb400fa6be6226ed29da7d544/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/cec489395f10eaf32edd5931540d6c28d0ba0e06d2482de5e4f6770ba249e6aa/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/ff3d3ae2b26f43e96f1b8e61c29481f70fa2fb106a09f8b63393a30920ba00c1/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/dae3998a33e98103ceb0ba3916b1ff89304f1ef3529597411b21e25a757eb71a/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/271beeb8d55a93ed0142deec33148b4c4742f7558f9120dc09f1e4ccbea6acdd/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/f2295d6417216ce2cbdb4bd89b654f1e25c4d461e73cb62298d56cadb3f42e07/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/cc826e69d7fd0d4ed430b7ce40f561affc50acf5d614043f580f9afb40ec15d7/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/5307f0a1be4923fea83e4bc1791f76ef24cedf2cc50f1422651c90f7be07264a/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/dd2c619a4b5ff3b4490d8452be07dbf69ec3e8de9624345d4d9c0201542a00f6/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/a3768a4301d99efc1f6d9cc19a9a6c2d73faa65ce9d71c4409cace77febb553e/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/f60818287df0db38be7d47a3619d5d12c2a91160aaa089d100f642c93bbaae08/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/37c8b2d31863a7043046ede7597507ba50982b386a6d4e4b36a8f33399189483/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/3a3f24ddb0feae3e50f1e676983eaf5ddf38f41cd360b7065e2f8de42fdb8ba5/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/0afb43428bef69d24fec1cc7b4e33e282741eef81b64a689c500bd2a94056f4a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-55:{mountpoint:/var/lib/containers/storage/overlay/f9fabe2f8f54bf5f1e8bb9e03938bf27a12ad237344e1e6cec1a60971e0ca182/merged major:0 minor:55 fsType:overlay blockSize:0} overlay_0-57:{mountpoint:/var/lib/containers/storage/overlay/8fc01791f7ee265e52d3464343cbbd50ab8476f7955943d0baed6a33e158e2d8/merged major:0 minor:57 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/b8d5c8d7c00da1abaeec2656977bec95be93fcb6fd961c8ba394ede9e23bd63d/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/76c316756c0158d6f9b48afe4898281a705e154a988c1bcbd7736a0011331144/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/c95c0d1a221b96ef3b007e89331d94ea50f61ca17cd2d1f9e8193794108c34ae/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/e5f1af1b0098920d1976416213d577e4fe9c84b9ab899cd81ca69e1df4c1c88a/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/9eab1f52e721ccf356b9c136b2d85dccd96815b38badbc64f80451b51e817203/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/6a177c1d43363fef93ed61bd5ed8fe9806d4af65a5eecc592b7d41c7628ef9e0/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/2847a406f2de0a49ef34a749edefbf9fa3078e4d8b6548694cbaf279c2a963ec/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 04 00:58:37.944162 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.941340 4852 manager.go:217] Machine: {Timestamp:2026-11-04 00:58:37.940901782 +0000 UTC m=+0.056011891 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098639 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2519dbc6858107bacce90bbc2fc5da SystemUUID:ec2519db-c685-8107-bacc-e90bbc2fc5da BootID:5a0d2f3f-4841-463d-849a-fc8f7d213cae Filesystems:[{Device:/run/containers/storage/overlay-containers/2c2ff0ae8b6107ced8dc11426662d452592f20733f317e1f9d573ff3f9f48ddb/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/116018e6f7a57844a290bbeafac250cf3eec2bd16bd247250c1589e0ee539b7c/userdata/shm DeviceMajor:0 DeviceMinor:103 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/55e1fd53-92c2-484e-9405-ac5af81ed61b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6eada63-a80f-4eff-8076-6f01b9efa23e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/847ad3d1-d945-4281-9a5b-15420289c70d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6eada63-a80f-4eff-8076-6f01b9efa23e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ac5e9e6b-7405-4c03-a149-4bf353209f24/volumes/kubernetes.io~projected/kube-api-access-xzwsb DeviceMajor:0 DeviceMinor:72 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/975903c9-9da2-4971-a50b-7bcc37df6b34/volumes/kubernetes.io~projected/kube-api-access-bdntp DeviceMajor:0 DeviceMinor:73 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5d50c29d-9104-4dd5-9499-7dd94594666b/volumes/kubernetes.io~projected/kube-api-access-65cwq DeviceMajor:0 DeviceMinor:114 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-199 DeviceMajor:0 DeviceMinor:199 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c/volumes/kubernetes.io~projected/kube-api-access-drqhw DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b0a4b0b861d58a2c7583fedabd3e2e24771e738f0e5ece34c94211c52375518/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-57 DeviceMajor:0 DeviceMinor:57 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6eada63-a80f-4eff-8076-6f01b9efa23e/volumes/kubernetes.io~projected/kube-api-access-j8dw2 DeviceMajor:0 DeviceMinor:69 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f6083261-f4cb-41ee-999d-8c5406473ff8/volumes/kubernetes.io~projected/kube-api-access-z8ms4 DeviceMajor:0 DeviceMinor:75 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/847ad3d1-d945-4281-9a5b-15420289c70d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4832f0e32b037011ba261e4a227f95b569e553145a1aec19d3bd81bebceef542/userdata/shm DeviceMajor:0 DeviceMinor:108 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ac7c69deda762ae8537940d4a40a5047556167936c3d3932d3fc617ce507b23d/userdata/shm DeviceMajor:0 DeviceMinor:100 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-110 DeviceMajor:0 DeviceMinor:110 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/55e1fd53-92c2-484e-9405-ac5af81ed61b/volumes/kubernetes.io~projected/kube-api-access-m8q28 DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c348450f2fa7f6112786c73979ebec84d5fbc267418d8b91e15825916c9eefd/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/277ddfb0de2eee6d0a584457c024d05cb0c5fa9ced843e754720beae14816406/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/928b0878ac73f0c4f75f2d9253aa831dae7a8d135a26d976dbd74a8cf4f1f552/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b0240716b1410bacd943746df2af03430f99980a3e0acc79aa62c401898a250/userdata/shm DeviceMajor:0 DeviceMinor:260 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/847ad3d1-d945-4281-9a5b-15420289c70d/volumes/kubernetes.io~projected/kube-api-access-j7s8t DeviceMajor:0 DeviceMinor:70 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec/volumes/kubernetes.io~projected/kube-api-access-llvc6 DeviceMajor:0 DeviceMinor:258 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bb2c2fb8-f28c-430b-8210-db7032a7073b/volumes/kubernetes.io~projected/kube-api-access-pdklr DeviceMajor:0 DeviceMinor:76 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97ca9fe8c890a540b9403fd94faf0edb071532c8f5e838a5a6b274bb9ed59f51/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/47fb874a-90bd-41f9-9a8c-0e920d789ca2/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:253 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-126 DeviceMajor:0 DeviceMinor:126 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-156 DeviceMajor:0 DeviceMinor:156 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-194 DeviceMajor:0 DeviceMinor:194 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5c23d2cad0687c62de401a72caf2927884cc276b0be1f6d294366179128eaa7/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f784deee-bbd7-4e50-94d1-7bb64fc61b91/volumes/kubernetes.io~projected/kube-api-access-htztj DeviceMajor:0 DeviceMinor:77 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-176 DeviceMajor:0 DeviceMinor:176 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66117eac-ae66-462e-8282-7ef04b047fbe/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/66117eac-ae66-462e-8282-7ef04b047fbe/volumes/kubernetes.io~projected/kube-api-access-8vt9h DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95ffdb116aae4604d44141563ca1e3486d047bf765addbbb136234608ae2b4c4/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/652f3da9f495dbf0ada3f6694492cdcaf14feec9c4f9038817871fec701a498a/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7cbe8a51-0318-450b-adb2-4abfe1097883/volumes/kubernetes.io~projected/kube-api-access-xttk5 DeviceMajor:0 DeviceMinor:68 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/402ccc5f-d755-4061-a96b-8ddc0495b711/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:71 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b097049413e19ac6bbccd2cd05485546e0b37baf228f313ac7f97a283096122/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3342c8d9052dfe9f3803d0db938c962dc58657ee79a9802b49e5be497a29772b/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bb2c2fb8-f28c-430b-8210-db7032a7073b/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:257 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-55 DeviceMajor:0 DeviceMinor:55 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-189 DeviceMajor:0 DeviceMinor:189 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/5d50c29d-9104-4dd5-9499-7dd94594666b/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:118 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5d50c29d-9104-4dd5-9499-7dd94594666b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:119 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af1da2809571589a0f36763d612938735298294e863cbf70f0576500fab216b5/userdata/shm DeviceMajor:0 DeviceMinor:120 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3fc4c1e4b8b2fb772b8257e1316188ed74a737fd023c3acf4d90b68c20be30c8/userdata/shm DeviceMajor:0 DeviceMinor:267 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/565a71a6-1ce7-4078-90fc-f2ccadaa0d17/volumes/kubernetes.io~projected/kube-api-access-npmf5 DeviceMajor:0 DeviceMinor:74 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:277ddfb0de2eee6 MacAddress:6e:bb:fa:7e:4f:a3 Speed:10000 Mtu:8901} {Name:3fc4c1e4b8b2fb7 MacAddress:0e:24:94:d6:6c:ac Speed:10000 Mtu:8901} {Name:8c348450f2fa7f6 MacAddress:92:43:85:e7:08:61 Speed:10000 Mtu:8901} {Name:97ca9fe8c890a54 MacAddress:e6:9e:d4:c6:02:ec Speed:10000 Mtu:8901} {Name:9b0240716b1410b MacAddress:fa:a7:f9:f7:50:1b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ed:ff:7c:eb Speed:0 Mtu:9001} {Name:br-int MacAddress:de:76:c4:9f:71:74 Speed:0 Mtu:8901} {Name:c5c23d2cad0687c MacAddress:b2:2f:7f:46:30:11 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ed:ff:7c:eb Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ea:68:1d:51:68:5f Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:42:33:c3:e0:9d:f0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 00:58:37.944162 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.941620 4852 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 00:58:37.944162 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.941657 4852 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 00:58:37.944162 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.941898 4852 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 00:58:37.944162 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.941924 4852 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-241.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 00:58:37.944162 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.942048 4852 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 00:58:37.944162 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.942054 4852 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 00:58:37.944162 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.942067 4852 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 00:58:37.944162 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.942081 4852 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 00:58:37.944162 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.942145 4852 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 00:58:37.944162 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.942298 4852 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 00:58:37.944162 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.942391 4852 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 00:58:37.944162 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.942432 4852 kubelet.go:493] "Attempting to sync node with API server" Nov 04 00:58:37.944162 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.942443 4852 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 00:58:37.944162 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.942451 4852 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 00:58:37.944162 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.942460 4852 kubelet.go:405] "Adding apiserver pod source" Nov 04 00:58:37.944162 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.942467 4852 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 00:58:37.944162 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.943046 4852 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 00:58:37.944162 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.943054 4852 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 00:58:37.944162 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.943554 4852 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 00:58:37.946824 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.946808 4852 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 00:58:37.946824 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.946824 4852 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 00:58:37.946955 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.946932 4852 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 04 00:58:37.947170 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.947154 4852 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 00:58:37.947235 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.947196 4852 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 00:58:37.947319 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.947307 4852 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 00:58:37.947371 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.947326 4852 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 00:58:37.947371 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.947337 4852 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 00:58:37.947371 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.947346 4852 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 00:58:37.947371 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.947355 4852 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 00:58:37.947371 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.947365 4852 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 00:58:37.947702 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.947374 4852 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 00:58:37.947702 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.947382 4852 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 00:58:37.947702 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.947393 4852 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 00:58:37.947702 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.947403 4852 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 00:58:37.947702 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.947416 4852 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 00:58:37.947702 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.947430 4852 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 00:58:37.947702 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.947463 4852 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 00:58:37.947702 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.947473 4852 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 00:58:37.950709 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.950695 4852 server.go:1263] "Started kubelet" Nov 04 00:58:37.950839 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.950787 4852 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 00:58:37.950946 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.950902 4852 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 00:58:37.950996 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.950966 4852 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 00:58:37.951142 ip-10-0-140-241 systemd[1]: Started Kubernetes Kubelet. Nov 04 00:58:37.952246 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.951228 4852 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 00:58:37.952246 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.951888 4852 server.go:317] "Adding debug handlers to kubelet server" Nov 04 00:58:37.953623 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.953606 4852 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 00:58:37.953835 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.953817 4852 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 00:58:37.954405 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.954123 4852 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 00:53:00 +0000 UTC" deadline="2028-04-16 08:29:07.63248443 +0000 UTC" Nov 04 00:58:37.954405 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.954159 4852 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12703h30m29.678328096s" Nov 04 00:58:37.956298 ip-10-0-140-241 kubenswrapper[4852]: E1104 00:58:37.955640 4852 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-241.us-east-2.compute.internal\" not found" Nov 04 00:58:37.956298 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.955659 4852 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 00:58:37.956298 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.955674 4852 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 00:58:37.956298 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.955697 4852 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 00:58:37.958612 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.958546 4852 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.959987 4852 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.960605 4852 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961211 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c" volumeName="kubernetes.io/projected/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-kube-api-access-drqhw" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961256 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d50c29d-9104-4dd5-9499-7dd94594666b" volumeName="kubernetes.io/configmap/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961270 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d50c29d-9104-4dd5-9499-7dd94594666b" volumeName="kubernetes.io/empty-dir/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-textfile" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961290 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cbe8a51-0318-450b-adb2-4abfe1097883" volumeName="kubernetes.io/empty-dir/7cbe8a51-0318-450b-adb2-4abfe1097883-tmp-dir" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961303 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55e1fd53-92c2-484e-9405-ac5af81ed61b" volumeName="kubernetes.io/empty-dir/55e1fd53-92c2-484e-9405-ac5af81ed61b-data-volume" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961324 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55e1fd53-92c2-484e-9405-ac5af81ed61b" volumeName="kubernetes.io/projected/55e1fd53-92c2-484e-9405-ac5af81ed61b-kube-api-access-m8q28" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961339 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="847ad3d1-d945-4281-9a5b-15420289c70d" volumeName="kubernetes.io/empty-dir/847ad3d1-d945-4281-9a5b-15420289c70d-tmp" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961352 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb2c2fb8-f28c-430b-8210-db7032a7073b" volumeName="kubernetes.io/projected/bb2c2fb8-f28c-430b-8210-db7032a7073b-kube-api-access-pdklr" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961374 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6eada63-a80f-4eff-8076-6f01b9efa23e" volumeName="kubernetes.io/configmap/c6eada63-a80f-4eff-8076-6f01b9efa23e-ovnkube-config" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961390 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6eada63-a80f-4eff-8076-6f01b9efa23e" volumeName="kubernetes.io/secret/c6eada63-a80f-4eff-8076-6f01b9efa23e-ovn-node-metrics-cert" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961412 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66117eac-ae66-462e-8282-7ef04b047fbe" volumeName="kubernetes.io/projected/66117eac-ae66-462e-8282-7ef04b047fbe-kube-api-access-8vt9h" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961428 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac5e9e6b-7405-4c03-a149-4bf353209f24" volumeName="kubernetes.io/configmap/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-daemon-config" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961449 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6083261-f4cb-41ee-999d-8c5406473ff8" volumeName="kubernetes.io/configmap/f6083261-f4cb-41ee-999d-8c5406473ff8-iptables-alerter-script" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961464 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6083261-f4cb-41ee-999d-8c5406473ff8" volumeName="kubernetes.io/projected/f6083261-f4cb-41ee-999d-8c5406473ff8-kube-api-access-z8ms4" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961491 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="402ccc5f-d755-4061-a96b-8ddc0495b711" volumeName="kubernetes.io/configmap/402ccc5f-d755-4061-a96b-8ddc0495b711-konnectivity-ca" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961511 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55e1fd53-92c2-484e-9405-ac5af81ed61b" volumeName="kubernetes.io/configmap/55e1fd53-92c2-484e-9405-ac5af81ed61b-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961523 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="565a71a6-1ce7-4078-90fc-f2ccadaa0d17" volumeName="kubernetes.io/projected/565a71a6-1ce7-4078-90fc-f2ccadaa0d17-kube-api-access-npmf5" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961544 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d50c29d-9104-4dd5-9499-7dd94594666b" volumeName="kubernetes.io/projected/5d50c29d-9104-4dd5-9499-7dd94594666b-kube-api-access-65cwq" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961581 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d50c29d-9104-4dd5-9499-7dd94594666b" volumeName="kubernetes.io/secret/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961603 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="847ad3d1-d945-4281-9a5b-15420289c70d" volumeName="kubernetes.io/projected/847ad3d1-d945-4281-9a5b-15420289c70d-kube-api-access-j7s8t" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961620 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6eada63-a80f-4eff-8076-6f01b9efa23e" volumeName="kubernetes.io/configmap/c6eada63-a80f-4eff-8076-6f01b9efa23e-ovnkube-script-lib" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961638 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f784deee-bbd7-4e50-94d1-7bb64fc61b91" volumeName="kubernetes.io/configmap/f784deee-bbd7-4e50-94d1-7bb64fc61b91-cni-binary-copy" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961656 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47fb874a-90bd-41f9-9a8c-0e920d789ca2" volumeName="kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961671 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66117eac-ae66-462e-8282-7ef04b047fbe" volumeName="kubernetes.io/secret/66117eac-ae66-462e-8282-7ef04b047fbe-cert" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961691 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="847ad3d1-d945-4281-9a5b-15420289c70d" volumeName="kubernetes.io/empty-dir/847ad3d1-d945-4281-9a5b-15420289c70d-etc-tuned" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961705 4852 factory.go:55] Registering systemd factory Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961723 4852 factory.go:223] Registration of the systemd container factory successfully Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961704 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="975903c9-9da2-4971-a50b-7bcc37df6b34" volumeName="kubernetes.io/projected/975903c9-9da2-4971-a50b-7bcc37df6b34-kube-api-access-bdntp" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961815 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac5e9e6b-7405-4c03-a149-4bf353209f24" volumeName="kubernetes.io/projected/ac5e9e6b-7405-4c03-a149-4bf353209f24-kube-api-access-xzwsb" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961843 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6eada63-a80f-4eff-8076-6f01b9efa23e" volumeName="kubernetes.io/configmap/c6eada63-a80f-4eff-8076-6f01b9efa23e-env-overrides" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961913 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6eada63-a80f-4eff-8076-6f01b9efa23e" volumeName="kubernetes.io/projected/c6eada63-a80f-4eff-8076-6f01b9efa23e-kube-api-access-j8dw2" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961935 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f784deee-bbd7-4e50-94d1-7bb64fc61b91" volumeName="kubernetes.io/configmap/f784deee-bbd7-4e50-94d1-7bb64fc61b91-cni-sysctl-allowlist" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.961950 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c" volumeName="kubernetes.io/secret/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-metrics-tls" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.962001 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cbe8a51-0318-450b-adb2-4abfe1097883" volumeName="kubernetes.io/projected/7cbe8a51-0318-450b-adb2-4abfe1097883-kube-api-access-xttk5" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.962096 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac5e9e6b-7405-4c03-a149-4bf353209f24" volumeName="kubernetes.io/configmap/ac5e9e6b-7405-4c03-a149-4bf353209f24-cni-binary-copy" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.962128 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f784deee-bbd7-4e50-94d1-7bb64fc61b91" volumeName="kubernetes.io/projected/f784deee-bbd7-4e50-94d1-7bb64fc61b91-kube-api-access-htztj" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.962157 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c" volumeName="kubernetes.io/empty-dir/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-tmp-dir" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.962171 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="402ccc5f-d755-4061-a96b-8ddc0495b711" volumeName="kubernetes.io/secret/402ccc5f-d755-4061-a96b-8ddc0495b711-agent-certs" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.962187 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="565a71a6-1ce7-4078-90fc-f2ccadaa0d17" volumeName="kubernetes.io/configmap/565a71a6-1ce7-4078-90fc-f2ccadaa0d17-serviceca" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.962208 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec" volumeName="kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.962221 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55e1fd53-92c2-484e-9405-ac5af81ed61b" volumeName="kubernetes.io/secret/55e1fd53-92c2-484e-9405-ac5af81ed61b-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.962240 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d50c29d-9104-4dd5-9499-7dd94594666b" volumeName="kubernetes.io/configmap/5d50c29d-9104-4dd5-9499-7dd94594666b-metrics-client-ca" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.962256 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d50c29d-9104-4dd5-9499-7dd94594666b" volumeName="kubernetes.io/secret/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-tls" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.962270 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb2c2fb8-f28c-430b-8210-db7032a7073b" volumeName="kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.962288 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f784deee-bbd7-4e50-94d1-7bb64fc61b91" volumeName="kubernetes.io/configmap/f784deee-bbd7-4e50-94d1-7bb64fc61b91-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.962305 4852 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c" volumeName="kubernetes.io/configmap/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-config-volume" seLinuxMountContext="" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.962314 4852 reconstruct.go:97] "Volume reconstruction finished" Nov 04 00:58:37.962542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.962319 4852 reconciler.go:29] "Reconciler: start to sync state" Nov 04 00:58:37.964951 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.963792 4852 factory.go:153] Registering CRI-O factory Nov 04 00:58:37.964951 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.963810 4852 factory.go:223] Registration of the crio container factory successfully Nov 04 00:58:37.964951 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.963886 4852 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 00:58:37.964951 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.963911 4852 factory.go:103] Registering Raw factory Nov 04 00:58:37.964951 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.963926 4852 manager.go:1199] Started watching for new ooms in manager Nov 04 00:58:37.964951 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.964265 4852 manager.go:319] Starting recovery of all containers Nov 04 00:58:37.965980 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.965955 4852 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 04 00:58:37.980606 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.980556 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 00:58:37.982292 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.982273 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 00:58:37.985119 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.985069 4852 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 00:58:37.986712 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.986697 4852 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 00:58:37.986851 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.986720 4852 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 00:58:37.986851 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.986738 4852 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 00:58:37.986851 ip-10-0-140-241 kubenswrapper[4852]: E1104 00:58:37.986783 4852 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 04 00:58:37.990190 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:37.990169 4852 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:58:38.012202 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.012173 4852 generic.go:356] "Generic (PLEG): container finished" podID="5d50c29d-9104-4dd5-9499-7dd94594666b" containerID="fc11b4ec5655633f57e5583270dfceea5e62a822dfdaa0d87fbbcedb6dbc3adc" exitCode=0 Nov 04 00:58:38.015728 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.015703 4852 generic.go:356] "Generic (PLEG): container finished" podID="f784deee-bbd7-4e50-94d1-7bb64fc61b91" containerID="dfd9c596718db5027553ea47a0f1b27cd0deb96690660fa8e3f2e99be8f4683c" exitCode=0 Nov 04 00:58:38.015831 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.015730 4852 generic.go:356] "Generic (PLEG): container finished" podID="f784deee-bbd7-4e50-94d1-7bb64fc61b91" containerID="8c2d34649e814236c82d3b122c5a1da9680fd3acf1d8e99c8713f9e3761053bc" exitCode=0 Nov 04 00:58:38.015831 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.015742 4852 generic.go:356] "Generic (PLEG): container finished" podID="f784deee-bbd7-4e50-94d1-7bb64fc61b91" containerID="4a20ec771a7fc8287cd41f44de4823dcfda6dd91a4cb7f36411c0792b9d3e8f8" exitCode=0 Nov 04 00:58:38.015831 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.015751 4852 generic.go:356] "Generic (PLEG): container finished" podID="f784deee-bbd7-4e50-94d1-7bb64fc61b91" containerID="67ec8c3003a1bddffdf49ac5f78b5c1393e9702b83f3014d3d28d8cb79618ee4" exitCode=0 Nov 04 00:58:38.015831 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.015758 4852 generic.go:356] "Generic (PLEG): container finished" podID="f784deee-bbd7-4e50-94d1-7bb64fc61b91" containerID="ae8657394b127fdcc8096574f941d34e8a30f83f69afea865a9355275be81005" exitCode=0 Nov 04 00:58:38.015831 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.015766 4852 generic.go:356] "Generic (PLEG): container finished" podID="f784deee-bbd7-4e50-94d1-7bb64fc61b91" containerID="5a9107488220c7b7bfa765b47cc3e38765b2d0c8c1ce85297ecbe30fac04d1c0" exitCode=0 Nov 04 00:58:38.019756 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.019737 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 00:58:38.020117 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.020096 4852 generic.go:356] "Generic (PLEG): container finished" podID="c6eada63-a80f-4eff-8076-6f01b9efa23e" containerID="21ce13a10350e0cfc715625dc822391a209874e65b247efeca0d556a99a48bcb" exitCode=1 Nov 04 00:58:38.026262 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.026239 4852 generic.go:356] "Generic (PLEG): container finished" podID="f31320b40fb2fb40491617f2b2422d38" containerID="4ae69cfd44ebd6c7ccc3e6476aaf952e8693a901e1e6226f89d0f0c3faa64b28" exitCode=0 Nov 04 00:58:38.041350 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.041335 4852 manager.go:324] Recovery completed Nov 04 00:58:38.057056 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.057038 4852 cpu_manager.go:226] "Starting" policy="none" Nov 04 00:58:38.057056 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.057052 4852 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 00:58:38.057148 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.057072 4852 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 00:58:38.057202 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.057181 4852 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 04 00:58:38.057237 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.057202 4852 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 04 00:58:38.057237 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.057218 4852 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 04 00:58:38.057237 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.057223 4852 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 04 00:58:38.057237 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.057229 4852 policy_none.go:50] "Start" Nov 04 00:58:38.057237 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.057237 4852 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 00:58:38.057384 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.057246 4852 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 00:58:38.057384 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.057361 4852 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 04 00:58:38.057384 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.057368 4852 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 04 00:58:38.057384 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.057373 4852 policy_none.go:44] "Start" Nov 04 00:58:38.061297 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.061275 4852 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 00:58:38.063773 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.063758 4852 manager.go:341] "Starting Device Plugin manager" Nov 04 00:58:38.064320 ip-10-0-140-241 kubenswrapper[4852]: E1104 00:58:38.063786 4852 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 00:58:38.064320 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.064320 4852 server.go:87] "Starting device plugin registration server" Nov 04 00:58:38.064557 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.064543 4852 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 00:58:38.064649 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.064556 4852 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 00:58:38.064649 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.064635 4852 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 00:58:38.064789 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.064739 4852 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 00:58:38.064856 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.064799 4852 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 00:58:38.064856 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.064807 4852 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 00:58:38.064947 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.064850 4852 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T00:58:38.064786773Z","UUID":"c1e0e472-88ab-4951-96a4-b77e84ea4957","Handler":null,"Name":"","Endpoint":""} Nov 04 00:58:38.066662 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.066617 4852 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 00:58:38.066744 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.066674 4852 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 00:58:38.067062 ip-10-0-140-241 kubenswrapper[4852]: E1104 00:58:38.067038 4852 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 00:58:38.086862 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.086831 4852 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal"] Nov 04 00:58:38.087337 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.087306 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" event={"ID":"f31320b40fb2fb40491617f2b2422d38","Type":"ContainerStarted","Data":"ba6501ed90a71c6188a16ab5392f0e919818e7459cb3ea99966cf5cb49712cf9"} Nov 04 00:58:38.087427 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.087344 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" event={"ID":"f31320b40fb2fb40491617f2b2422d38","Type":"ContainerDied","Data":"4ae69cfd44ebd6c7ccc3e6476aaf952e8693a901e1e6226f89d0f0c3faa64b28"} Nov 04 00:58:38.087427 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.087363 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" event={"ID":"f31320b40fb2fb40491617f2b2422d38","Type":"ContainerStarted","Data":"652f3da9f495dbf0ada3f6694492cdcaf14feec9c4f9038817871fec701a498a"} Nov 04 00:58:38.087427 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.087374 4852 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:58:38.087553 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.087380 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal" event={"ID":"cf6706eb85e3c341c82e407ba659da4b","Type":"ContainerStarted","Data":"012550e65193b9ba709455d17a7e4204dd8d2d1d452eefaa2eb517ed062ee1a4"} Nov 04 00:58:38.087553 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.087455 4852 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:58:38.087553 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.087470 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal" event={"ID":"cf6706eb85e3c341c82e407ba659da4b","Type":"ContainerStarted","Data":"1b0a4b0b861d58a2c7583fedabd3e2e24771e738f0e5ece34c94211c52375518"} Nov 04 00:58:38.098253 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.098220 4852 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:58:38.098314 ip-10-0-140-241 kubenswrapper[4852]: E1104 00:58:38.098256 4852 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:58:38.098770 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.098757 4852 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:58:38.098824 ip-10-0-140-241 kubenswrapper[4852]: E1104 00:58:38.098791 4852 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:58:38.165040 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.165023 4852 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:58:38.166040 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.166019 4852 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:58:38.166131 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.166051 4852 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:58:38.166131 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.166061 4852 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-241.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:58:38.166131 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.166063 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f31320b40fb2fb40491617f2b2422d38-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal\" (UID: \"f31320b40fb2fb40491617f2b2422d38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:58:38.166131 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.166092 4852 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:58:38.166270 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.166089 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31320b40fb2fb40491617f2b2422d38-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal\" (UID: \"f31320b40fb2fb40491617f2b2422d38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:58:38.166270 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.166214 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf6706eb85e3c341c82e407ba659da4b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal\" (UID: \"cf6706eb85e3c341c82e407ba659da4b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:58:38.180061 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.180047 4852 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:58:38.180121 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.180095 4852 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:58:38.267123 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.267098 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f31320b40fb2fb40491617f2b2422d38-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal\" (UID: \"f31320b40fb2fb40491617f2b2422d38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:58:38.267198 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.267128 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31320b40fb2fb40491617f2b2422d38-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal\" (UID: \"f31320b40fb2fb40491617f2b2422d38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:58:38.267198 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.267147 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf6706eb85e3c341c82e407ba659da4b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal\" (UID: \"cf6706eb85e3c341c82e407ba659da4b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:58:38.267270 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.267200 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f31320b40fb2fb40491617f2b2422d38-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal\" (UID: \"f31320b40fb2fb40491617f2b2422d38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:58:38.267270 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.267216 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31320b40fb2fb40491617f2b2422d38-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal\" (UID: \"f31320b40fb2fb40491617f2b2422d38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:58:38.267270 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.267200 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf6706eb85e3c341c82e407ba659da4b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal\" (UID: \"cf6706eb85e3c341c82e407ba659da4b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:58:38.943521 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.943486 4852 apiserver.go:52] "Watching apiserver" Nov 04 00:58:38.952235 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.952216 4852 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 00:58:38.952686 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.952668 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal","openshift-monitoring/node-exporter-vqnkh","openshift-dns/dns-default-z9szb","openshift-ingress-canary/ingress-canary-xw58g","openshift-network-diagnostics/network-check-target-b4k7w","openshift-multus/network-metrics-daemon-dsxl7","kube-system/kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal","openshift-multus/multus-dxbxk","openshift-insights/insights-runtime-extractor-n2w4k","kube-system/konnectivity-agent-8d5qr","kube-system/global-pull-secret-syncer-5v9xx","openshift-image-registry/node-ca-v8q47","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n","openshift-dns/node-resolver-7vrj7","openshift-network-operator/iptables-alerter-jlf57","openshift-cluster-node-tuning-operator/tuned-hlvm6","openshift-multus/multus-additional-cni-plugins-mlx4q","openshift-ovn-kubernetes/ovnkube-node-vst89"] Nov 04 00:58:38.957303 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.957279 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 00:58:38.957428 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.957403 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 00:58:38.957488 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.957418 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 00:58:38.958932 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.958908 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 00:58:38.959025 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.958932 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 00:58:38.959025 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.958956 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 00:58:38.959025 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.958963 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zx855\"" Nov 04 00:58:38.959025 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.958966 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 00:58:38.959223 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.959046 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sknxk\"" Nov 04 00:58:38.959223 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.959199 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 00:58:38.959318 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.959243 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 00:58:38.959366 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.959318 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nszlw\"" Nov 04 00:58:38.959619 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.959600 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4rmxc\"" Nov 04 00:58:38.959685 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.959619 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 00:58:38.959772 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.959754 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9jwd5\"" Nov 04 00:58:38.959821 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.959761 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 00:58:38.959821 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.959812 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 00:58:38.959927 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.959837 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 00:58:38.959927 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.959860 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 00:58:38.959927 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.959863 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 00:58:38.959927 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.959878 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 00:58:38.959927 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.959897 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 00:58:38.959927 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.959903 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 00:58:38.960184 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.959950 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q48lg\"" Nov 04 00:58:38.960184 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.960141 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 00:58:38.960279 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.960246 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ln4qv\"" Nov 04 00:58:38.960279 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.960265 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:58:38.960916 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.960901 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 00:58:38.961015 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.960982 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-xdt54\"" Nov 04 00:58:38.962223 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.962200 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zgrbh\"" Nov 04 00:58:38.962309 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.962212 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 00:58:38.962309 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.962238 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 00:58:38.962309 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.962293 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:58:38.962689 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.962670 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 00:58:38.962769 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.962681 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 00:58:38.962769 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.962708 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4k8lm\"" Nov 04 00:58:38.962971 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.962954 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 00:58:38.963091 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.963011 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-psr9r\"" Nov 04 00:58:38.963376 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.963364 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 00:58:38.963504 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.963494 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 00:58:38.963668 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.963655 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kqwbk\"" Nov 04 00:58:38.963974 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.963959 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 00:58:38.964072 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.964058 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 00:58:38.964356 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.964342 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 04 00:58:38.964897 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.964884 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 00:58:38.965076 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.965051 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 00:58:38.965494 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.965478 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 00:58:38.965597 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.965481 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 04 00:58:38.965597 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.965494 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 00:58:38.965597 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.965510 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5llll\"" Nov 04 00:58:38.965597 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.965541 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 04 00:58:38.965597 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.965548 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 04 00:58:38.966328 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.966309 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 00:58:38.966328 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.966325 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 00:58:38.966498 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.966333 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 00:58:38.966498 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.966334 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 00:58:38.966498 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.966348 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 00:58:38.966498 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.966364 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ldjm6\"" Nov 04 00:58:38.966498 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.966387 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pb69q\"" Nov 04 00:58:38.966498 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.966446 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 00:58:38.966498 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.966486 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 00:58:38.966711 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.966518 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cjss9\"" Nov 04 00:58:38.970967 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.970952 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-node-log\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.971010 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.970973 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f784deee-bbd7-4e50-94d1-7bb64fc61b91-system-cni-dir\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:38.971010 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.970994 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llvc6\" (UniqueName: \"kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6\") pod \"network-check-target-b4k7w\" (UID: \"0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec\") " pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:38.971101 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971010 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/847ad3d1-d945-4281-9a5b-15420289c70d-tmp\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:38.971101 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971025 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-sys-fs\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:38.971183 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971100 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f784deee-bbd7-4e50-94d1-7bb64fc61b91-cni-binary-copy\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:38.971225 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971204 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-sysconfig\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:38.971225 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971214 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/847ad3d1-d945-4281-9a5b-15420289c70d-tmp\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:38.971285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971232 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-host\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:38.971285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971252 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-var-lib-openvswitch\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.971342 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971306 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.971373 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971353 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6eada63-a80f-4eff-8076-6f01b9efa23e-ovnkube-config\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.971401 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971387 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/47fb874a-90bd-41f9-9a8c-0e920d789ca2-kubelet-config\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:38.971499 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971483 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-cni-bin\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.971529 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971512 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-cni-netd\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.971557 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971535 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6eada63-a80f-4eff-8076-6f01b9efa23e-ovnkube-config\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.971606 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971581 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f784deee-bbd7-4e50-94d1-7bb64fc61b91-cni-binary-copy\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:38.971635 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971605 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6eada63-a80f-4eff-8076-6f01b9efa23e-env-overrides\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.971635 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971624 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f784deee-bbd7-4e50-94d1-7bb64fc61b91-cnibin\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:38.971693 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971638 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-systemd\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:38.971693 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971663 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdntp\" (UniqueName: \"kubernetes.io/projected/975903c9-9da2-4971-a50b-7bcc37df6b34-kube-api-access-bdntp\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:38.971745 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971695 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6eada63-a80f-4eff-8076-6f01b9efa23e-ovnkube-script-lib\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.971776 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971745 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6eada63-a80f-4eff-8076-6f01b9efa23e-env-overrides\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.971804 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971774 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-kubernetes\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:38.971836 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971802 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-var-lib-kubelet\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:38.971836 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971827 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdklr\" (UniqueName: \"kubernetes.io/projected/bb2c2fb8-f28c-430b-8210-db7032a7073b-kube-api-access-pdklr\") pod \"network-metrics-daemon-dsxl7\" (UID: \"bb2c2fb8-f28c-430b-8210-db7032a7073b\") " pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:38.971897 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971860 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6eada63-a80f-4eff-8076-6f01b9efa23e-ovnkube-script-lib\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.971989 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971975 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-log-socket\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.972021 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.971999 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8dw2\" (UniqueName: \"kubernetes.io/projected/c6eada63-a80f-4eff-8076-6f01b9efa23e-kube-api-access-j8dw2\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.972021 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972014 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-sysctl-conf\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:38.972108 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972030 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:38.972108 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972045 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-run-openvswitch\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.972108 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972089 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-run-ovn\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.972251 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972135 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htztj\" (UniqueName: \"kubernetes.io/projected/f784deee-bbd7-4e50-94d1-7bb64fc61b91-kube-api-access-htztj\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:38.972251 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972165 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-sysctl-d\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:38.972251 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972192 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-run\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:38.972251 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972208 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:38.972251 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972235 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-etc-openvswitch\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.972449 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972256 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-socket-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:38.972449 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972272 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f784deee-bbd7-4e50-94d1-7bb64fc61b91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:38.972449 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972288 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-sys\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:38.972449 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972302 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-registration-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:38.972449 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972331 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-run-systemd\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.972449 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972358 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f784deee-bbd7-4e50-94d1-7bb64fc61b91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:38.972449 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972398 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/847ad3d1-d945-4281-9a5b-15420289c70d-etc-tuned\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:38.972449 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972419 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47fb874a-90bd-41f9-9a8c-0e920d789ca2-original-pull-secret\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:38.972745 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972467 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-kubelet\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.972745 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972488 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/847ad3d1-d945-4281-9a5b-15420289c70d-etc-tuned\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:38.972745 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972513 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6eada63-a80f-4eff-8076-6f01b9efa23e-ovn-node-metrics-cert\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.972745 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972543 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f784deee-bbd7-4e50-94d1-7bb64fc61b91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:38.972745 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972545 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-systemd-units\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.972745 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972605 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-modprobe-d\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:38.972745 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972637 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs\") pod \"network-metrics-daemon-dsxl7\" (UID: \"bb2c2fb8-f28c-430b-8210-db7032a7073b\") " pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:38.972745 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972660 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-etc-selinux\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:38.972745 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972695 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-run-netns\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.972745 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972704 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6eada63-a80f-4eff-8076-6f01b9efa23e-ovn-node-metrics-cert\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.972745 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972736 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7s8t\" (UniqueName: \"kubernetes.io/projected/847ad3d1-d945-4281-9a5b-15420289c70d-kube-api-access-j7s8t\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:38.973097 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972795 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2c2fb8-f28c-430b-8210-db7032a7073b-metrics-certs\") pod \"network-metrics-daemon-dsxl7\" (UID: \"bb2c2fb8-f28c-430b-8210-db7032a7073b\") " pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:38.973097 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972848 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-slash\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.973097 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972882 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-run-ovn-kubernetes\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.973097 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972906 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-lib-modules\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:38.973097 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972950 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f784deee-bbd7-4e50-94d1-7bb64fc61b91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:38.973097 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.972997 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-device-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:38.973097 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.973028 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/47fb874a-90bd-41f9-9a8c-0e920d789ca2-dbus\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:38.973299 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.973132 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f784deee-bbd7-4e50-94d1-7bb64fc61b91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:38.977257 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.977241 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llvc6\" (UniqueName: \"kubernetes.io/projected/0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec-kube-api-access-llvc6\") pod \"network-check-target-b4k7w\" (UID: \"0bfe7ded-d69c-49a5-9d70-b8aa3067e4ec\") " pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:38.978658 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.978638 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdklr\" (UniqueName: \"kubernetes.io/projected/bb2c2fb8-f28c-430b-8210-db7032a7073b-kube-api-access-pdklr\") pod \"network-metrics-daemon-dsxl7\" (UID: \"bb2c2fb8-f28c-430b-8210-db7032a7073b\") " pod="openshift-multus/network-metrics-daemon-dsxl7" Nov 04 00:58:38.978727 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.978714 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdntp\" (UniqueName: \"kubernetes.io/projected/975903c9-9da2-4971-a50b-7bcc37df6b34-kube-api-access-bdntp\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:38.981305 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.981281 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8dw2\" (UniqueName: \"kubernetes.io/projected/c6eada63-a80f-4eff-8076-6f01b9efa23e-kube-api-access-j8dw2\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:38.981377 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.981339 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7s8t\" (UniqueName: \"kubernetes.io/projected/847ad3d1-d945-4281-9a5b-15420289c70d-kube-api-access-j7s8t\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:38.981469 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:38.981454 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htztj\" (UniqueName: \"kubernetes.io/projected/f784deee-bbd7-4e50-94d1-7bb64fc61b91-kube-api-access-htztj\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:39.036800 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.036774 4852 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:58:39.037287 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.037266 4852 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:58:39.049303 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.049277 4852 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:58:39.049389 ip-10-0-140-241 kubenswrapper[4852]: E1104 00:58:39.049325 4852 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:58:39.050964 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.050950 4852 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:58:39.051042 ip-10-0-140-241 kubenswrapper[4852]: E1104 00:58:39.050982 4852 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-241.us-east-2.compute.internal" Nov 04 00:58:39.056186 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.056166 4852 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 00:58:39.073717 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.073699 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-cnibin\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.073795 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.073722 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-run-netns\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.073795 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.073736 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-conf-dir\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.073795 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.073753 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-daemon-config\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.073795 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.073767 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-run-multus-certs\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.073795 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.073782 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-device-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:39.073958 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.073848 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-device-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:39.073958 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.073858 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/47fb874a-90bd-41f9-9a8c-0e920d789ca2-dbus\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:39.073958 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.073899 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-node-log\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.073958 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.073924 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f784deee-bbd7-4e50-94d1-7bb64fc61b91-system-cni-dir\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:39.073958 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.073943 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/47fb874a-90bd-41f9-9a8c-0e920d789ca2-dbus\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:39.074124 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.073961 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-node-log\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.074124 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.073981 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f784deee-bbd7-4e50-94d1-7bb64fc61b91-system-cni-dir\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:39.074124 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074017 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-socket-dir-parent\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.074124 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074037 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6083261-f4cb-41ee-999d-8c5406473ff8-iptables-alerter-script\") pod \"iptables-alerter-jlf57\" (UID: \"f6083261-f4cb-41ee-999d-8c5406473ff8\") " pod="openshift-network-operator/iptables-alerter-jlf57" Nov 04 00:58:39.074124 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074061 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-sys-fs\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:39.074124 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074085 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-daemon-config\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.074124 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074103 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-sysconfig\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:39.074313 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074132 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-host\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:39.074313 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074148 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-sys-fs\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:39.074313 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074157 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npmf5\" (UniqueName: \"kubernetes.io/projected/565a71a6-1ce7-4078-90fc-f2ccadaa0d17-kube-api-access-npmf5\") pod \"node-ca-v8q47\" (UID: \"565a71a6-1ce7-4078-90fc-f2ccadaa0d17\") " pod="openshift-image-registry/node-ca-v8q47" Nov 04 00:58:39.074313 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074183 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-tmp-dir\") pod \"dns-default-z9szb\" (UID: \"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c\") " pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:39.074313 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074196 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-host\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:39.074313 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074199 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-sysconfig\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:39.074313 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074207 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drqhw\" (UniqueName: \"kubernetes.io/projected/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-kube-api-access-drqhw\") pod \"dns-default-z9szb\" (UID: \"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c\") " pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:39.074313 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074276 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6083261-f4cb-41ee-999d-8c5406473ff8-iptables-alerter-script\") pod \"iptables-alerter-jlf57\" (UID: \"f6083261-f4cb-41ee-999d-8c5406473ff8\") " pod="openshift-network-operator/iptables-alerter-jlf57" Nov 04 00:58:39.074313 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074308 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-var-lib-openvswitch\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.074546 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074340 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-tmp-dir\") pod \"dns-default-z9szb\" (UID: \"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c\") " pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:39.074546 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074401 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-var-lib-openvswitch\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.074546 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074400 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.074546 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074439 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/565a71a6-1ce7-4078-90fc-f2ccadaa0d17-serviceca\") pod \"node-ca-v8q47\" (UID: \"565a71a6-1ce7-4078-90fc-f2ccadaa0d17\") " pod="openshift-image-registry/node-ca-v8q47" Nov 04 00:58:39.074546 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074454 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d50c29d-9104-4dd5-9499-7dd94594666b-root\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:39.074546 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074454 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.074546 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074468 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55e1fd53-92c2-484e-9405-ac5af81ed61b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:39.074546 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074487 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/47fb874a-90bd-41f9-9a8c-0e920d789ca2-kubelet-config\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:39.074546 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074511 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-wtmp\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:39.074546 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074517 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/47fb874a-90bd-41f9-9a8c-0e920d789ca2-kubelet-config\") pod \"global-pull-secret-syncer-5v9xx\" (UID: \"47fb874a-90bd-41f9-9a8c-0e920d789ca2\") " pod="kube-system/global-pull-secret-syncer-5v9xx" Nov 04 00:58:39.074546 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074530 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac5e9e6b-7405-4c03-a149-4bf353209f24-cni-binary-copy\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.074894 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074596 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55e1fd53-92c2-484e-9405-ac5af81ed61b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:39.074894 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074682 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-cni-bin\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.074894 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074713 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac5e9e6b-7405-4c03-a149-4bf353209f24-cni-binary-copy\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.074894 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074744 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-cni-netd\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.074894 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074756 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55e1fd53-92c2-484e-9405-ac5af81ed61b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:39.074894 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074779 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-cni-bin\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.074894 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074787 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-cni-netd\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.074894 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074793 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/565a71a6-1ce7-4078-90fc-f2ccadaa0d17-serviceca\") pod \"node-ca-v8q47\" (UID: \"565a71a6-1ce7-4078-90fc-f2ccadaa0d17\") " pod="openshift-image-registry/node-ca-v8q47" Nov 04 00:58:39.074894 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074815 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f784deee-bbd7-4e50-94d1-7bb64fc61b91-cnibin\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:39.074894 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074822 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55e1fd53-92c2-484e-9405-ac5af81ed61b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:39.074894 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074839 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-systemd\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:39.074894 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074854 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f784deee-bbd7-4e50-94d1-7bb64fc61b91-cnibin\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:39.074894 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074887 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-config-volume\") pod \"dns-default-z9szb\" (UID: \"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c\") " pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:39.075285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074909 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-var-lib-kubelet\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.075285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074948 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-systemd\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:39.075285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.074959 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-kubernetes\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:39.075285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075060 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-kubernetes\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:39.075285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075053 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-var-lib-kubelet\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:39.075285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075069 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-config-volume\") pod \"dns-default-z9szb\" (UID: \"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c\") " pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:39.075285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075084 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55e1fd53-92c2-484e-9405-ac5af81ed61b-data-volume\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:39.075285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075090 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-var-lib-kubelet\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:39.075285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075101 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8q28\" (UniqueName: \"kubernetes.io/projected/55e1fd53-92c2-484e-9405-ac5af81ed61b-kube-api-access-m8q28\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:39.075285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075117 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-log-socket\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.075285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075129 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55e1fd53-92c2-484e-9405-ac5af81ed61b-data-volume\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:39.075285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075131 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-sysctl-conf\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:39.075285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075178 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:39.075285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075198 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-run-openvswitch\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.075285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075215 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-run-ovn\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.075285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075238 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-sysctl-d\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:39.075285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075245 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-sysctl-conf\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:39.075285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075153 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-log-socket\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.075285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075272 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-run\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:39.075285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075285 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-run-ovn\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.075285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075267 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-run-openvswitch\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.075285 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075293 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/402ccc5f-d755-4061-a96b-8ddc0495b711-konnectivity-ca\") pod \"konnectivity-agent-8d5qr\" (UID: \"402ccc5f-d755-4061-a96b-8ddc0495b711\") " pod="kube-system/konnectivity-agent-8d5qr" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075318 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-etc-openvswitch\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075269 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075349 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-var-lib-cni-multus\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075355 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-sysctl-d\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075323 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-run\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075376 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-etc-openvswitch\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075395 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7cbe8a51-0318-450b-adb2-4abfe1097883-hosts-file\") pod \"node-resolver-7vrj7\" (UID: \"7cbe8a51-0318-450b-adb2-4abfe1097883\") " pod="openshift-dns/node-resolver-7vrj7" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075419 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-socket-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075459 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-socket-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075484 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f784deee-bbd7-4e50-94d1-7bb64fc61b91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075503 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-sys\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075518 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/565a71a6-1ce7-4078-90fc-f2ccadaa0d17-host\") pod \"node-ca-v8q47\" (UID: \"565a71a6-1ce7-4078-90fc-f2ccadaa0d17\") " pod="openshift-image-registry/node-ca-v8q47" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075534 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075534 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f784deee-bbd7-4e50-94d1-7bb64fc61b91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mlx4q\" (UID: \"f784deee-bbd7-4e50-94d1-7bb64fc61b91\") " pod="openshift-multus/multus-additional-cni-plugins-mlx4q" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075555 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66117eac-ae66-462e-8282-7ef04b047fbe-cert\") pod \"ingress-canary-xw58g\" (UID: \"66117eac-ae66-462e-8282-7ef04b047fbe\") " pod="openshift-ingress-canary/ingress-canary-xw58g" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075597 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/402ccc5f-d755-4061-a96b-8ddc0495b711-konnectivity-ca\") pod \"konnectivity-agent-8d5qr\" (UID: \"402ccc5f-d755-4061-a96b-8ddc0495b711\") " pod="kube-system/konnectivity-agent-8d5qr" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075613 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-sys\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075618 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-var-lib-cni-bin\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075651 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-hostroot\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075669 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-registration-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075697 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-run-systemd\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075706 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-registration-dir\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075725 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-textfile\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075742 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-tls\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075757 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65cwq\" (UniqueName: \"kubernetes.io/projected/5d50c29d-9104-4dd5-9499-7dd94594666b-kube-api-access-65cwq\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075772 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzwsb\" (UniqueName: \"kubernetes.io/projected/ac5e9e6b-7405-4c03-a149-4bf353209f24-kube-api-access-xzwsb\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075775 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66117eac-ae66-462e-8282-7ef04b047fbe-cert\") pod \"ingress-canary-xw58g\" (UID: \"66117eac-ae66-462e-8282-7ef04b047fbe\") " pod="openshift-ingress-canary/ingress-canary-xw58g" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075795 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-kubelet\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075819 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-metrics-tls\") pod \"dns-default-z9szb\" (UID: \"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c\") " pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075824 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075830 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-run-systemd\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075828 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-textfile\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075846 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-etc-kubernetes\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075865 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-kubelet\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075870 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xttk5\" (UniqueName: \"kubernetes.io/projected/7cbe8a51-0318-450b-adb2-4abfe1097883-kube-api-access-xttk5\") pod \"node-resolver-7vrj7\" (UID: \"7cbe8a51-0318-450b-adb2-4abfe1097883\") " pod="openshift-dns/node-resolver-7vrj7" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075895 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55e1fd53-92c2-484e-9405-ac5af81ed61b-crio-socket\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075919 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-systemd-units\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075964 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-systemd-units\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.075960 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-modprobe-d\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076001 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-tls\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076004 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6083261-f4cb-41ee-999d-8c5406473ff8-host-slash\") pod \"iptables-alerter-jlf57\" (UID: \"f6083261-f4cb-41ee-999d-8c5406473ff8\") " pod="openshift-network-operator/iptables-alerter-jlf57" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076016 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-metrics-tls\") pod \"dns-default-z9szb\" (UID: \"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c\") " pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076046 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-etc-modprobe-d\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076051 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8ms4\" (UniqueName: \"kubernetes.io/projected/f6083261-f4cb-41ee-999d-8c5406473ff8-kube-api-access-z8ms4\") pod \"iptables-alerter-jlf57\" (UID: \"f6083261-f4cb-41ee-999d-8c5406473ff8\") " pod="openshift-network-operator/iptables-alerter-jlf57" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076075 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/402ccc5f-d755-4061-a96b-8ddc0495b711-agent-certs\") pod \"konnectivity-agent-8d5qr\" (UID: \"402ccc5f-d755-4061-a96b-8ddc0495b711\") " pod="kube-system/konnectivity-agent-8d5qr" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076098 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-etc-selinux\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:39.076096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076113 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-run-netns\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.077530 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076131 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-run-k8s-cni-cncf-io\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.077530 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076156 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cbe8a51-0318-450b-adb2-4abfe1097883-tmp-dir\") pod \"node-resolver-7vrj7\" (UID: \"7cbe8a51-0318-450b-adb2-4abfe1097883\") " pod="openshift-dns/node-resolver-7vrj7" Nov 04 00:58:39.077530 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076179 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/975903c9-9da2-4971-a50b-7bcc37df6b34-etc-selinux\") pod \"aws-ebs-csi-driver-node-dkc9n\" (UID: \"975903c9-9da2-4971-a50b-7bcc37df6b34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dkc9n" Nov 04 00:58:39.077530 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076183 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-slash\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.077530 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076217 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-slash\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.077530 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076218 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-run-ovn-kubernetes\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.077530 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076182 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-run-netns\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.077530 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076245 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-lib-modules\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:39.077530 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076252 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cbe8a51-0318-450b-adb2-4abfe1097883-tmp-dir\") pod \"node-resolver-7vrj7\" (UID: \"7cbe8a51-0318-450b-adb2-4abfe1097883\") " pod="openshift-dns/node-resolver-7vrj7" Nov 04 00:58:39.077530 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076268 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d50c29d-9104-4dd5-9499-7dd94594666b-sys\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:39.077530 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076264 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6eada63-a80f-4eff-8076-6f01b9efa23e-host-run-ovn-kubernetes\") pod \"ovnkube-node-vst89\" (UID: \"c6eada63-a80f-4eff-8076-6f01b9efa23e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:39.077530 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076294 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-accelerators-collector-config\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:39.077530 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076299 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/402ccc5f-d755-4061-a96b-8ddc0495b711-agent-certs\") pod \"konnectivity-agent-8d5qr\" (UID: \"402ccc5f-d755-4061-a96b-8ddc0495b711\") " pod="kube-system/konnectivity-agent-8d5qr" Nov 04 00:58:39.077530 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076338 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vt9h\" (UniqueName: \"kubernetes.io/projected/66117eac-ae66-462e-8282-7ef04b047fbe-kube-api-access-8vt9h\") pod \"ingress-canary-xw58g\" (UID: \"66117eac-ae66-462e-8282-7ef04b047fbe\") " pod="openshift-ingress-canary/ingress-canary-xw58g" Nov 04 00:58:39.077530 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076357 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/847ad3d1-d945-4281-9a5b-15420289c70d-lib-modules\") pod \"tuned-hlvm6\" (UID: \"847ad3d1-d945-4281-9a5b-15420289c70d\") " pod="openshift-cluster-node-tuning-operator/tuned-hlvm6" Nov 04 00:58:39.077530 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076366 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-cni-dir\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.077530 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076408 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d50c29d-9104-4dd5-9499-7dd94594666b-metrics-client-ca\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:39.077530 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076446 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-system-cni-dir\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.077530 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076496 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-accelerators-collector-config\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:39.077530 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.076619 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d50c29d-9104-4dd5-9499-7dd94594666b-metrics-client-ca\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:39.081743 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.081726 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drqhw\" (UniqueName: \"kubernetes.io/projected/1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c-kube-api-access-drqhw\") pod \"dns-default-z9szb\" (UID: \"1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c\") " pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:39.082985 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.082961 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8ms4\" (UniqueName: \"kubernetes.io/projected/f6083261-f4cb-41ee-999d-8c5406473ff8-kube-api-access-z8ms4\") pod \"iptables-alerter-jlf57\" (UID: \"f6083261-f4cb-41ee-999d-8c5406473ff8\") " pod="openshift-network-operator/iptables-alerter-jlf57" Nov 04 00:58:39.083413 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.083392 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xttk5\" (UniqueName: \"kubernetes.io/projected/7cbe8a51-0318-450b-adb2-4abfe1097883-kube-api-access-xttk5\") pod \"node-resolver-7vrj7\" (UID: \"7cbe8a51-0318-450b-adb2-4abfe1097883\") " pod="openshift-dns/node-resolver-7vrj7" Nov 04 00:58:39.083475 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.083396 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65cwq\" (UniqueName: \"kubernetes.io/projected/5d50c29d-9104-4dd5-9499-7dd94594666b-kube-api-access-65cwq\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:39.083552 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.083491 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzwsb\" (UniqueName: \"kubernetes.io/projected/ac5e9e6b-7405-4c03-a149-4bf353209f24-kube-api-access-xzwsb\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.083552 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.083489 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8q28\" (UniqueName: \"kubernetes.io/projected/55e1fd53-92c2-484e-9405-ac5af81ed61b-kube-api-access-m8q28\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:39.083552 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.083540 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npmf5\" (UniqueName: \"kubernetes.io/projected/565a71a6-1ce7-4078-90fc-f2ccadaa0d17-kube-api-access-npmf5\") pod \"node-ca-v8q47\" (UID: \"565a71a6-1ce7-4078-90fc-f2ccadaa0d17\") " pod="openshift-image-registry/node-ca-v8q47" Nov 04 00:58:39.083678 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.083664 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vt9h\" (UniqueName: \"kubernetes.io/projected/66117eac-ae66-462e-8282-7ef04b047fbe-kube-api-access-8vt9h\") pod \"ingress-canary-xw58g\" (UID: \"66117eac-ae66-462e-8282-7ef04b047fbe\") " pod="openshift-ingress-canary/ingress-canary-xw58g" Nov 04 00:58:39.177738 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.177711 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-etc-kubernetes\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.177738 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.177740 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55e1fd53-92c2-484e-9405-ac5af81ed61b-crio-socket\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:39.177872 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.177755 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6083261-f4cb-41ee-999d-8c5406473ff8-host-slash\") pod \"iptables-alerter-jlf57\" (UID: \"f6083261-f4cb-41ee-999d-8c5406473ff8\") " pod="openshift-network-operator/iptables-alerter-jlf57" Nov 04 00:58:39.177872 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.177780 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-run-k8s-cni-cncf-io\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.177872 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.177803 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d50c29d-9104-4dd5-9499-7dd94594666b-sys\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:39.177872 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.177829 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55e1fd53-92c2-484e-9405-ac5af81ed61b-crio-socket\") pod \"insights-runtime-extractor-n2w4k\" (UID: \"55e1fd53-92c2-484e-9405-ac5af81ed61b\") " pod="openshift-insights/insights-runtime-extractor-n2w4k" Nov 04 00:58:39.177872 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.177842 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-etc-kubernetes\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.177872 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.177866 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-cni-dir\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.178099 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.177870 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-run-k8s-cni-cncf-io\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.178099 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.177843 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6083261-f4cb-41ee-999d-8c5406473ff8-host-slash\") pod \"iptables-alerter-jlf57\" (UID: \"f6083261-f4cb-41ee-999d-8c5406473ff8\") " pod="openshift-network-operator/iptables-alerter-jlf57" Nov 04 00:58:39.178099 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.177849 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d50c29d-9104-4dd5-9499-7dd94594666b-sys\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:39.178099 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.177896 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-system-cni-dir\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.178099 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.177924 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-cnibin\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.178099 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.177937 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-system-cni-dir\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.178099 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.177940 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-run-netns\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.178099 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.177922 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-cni-dir\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.178099 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.177963 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-cnibin\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.178099 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.177965 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-conf-dir\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.178099 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.177991 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-run-netns\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.178099 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.178001 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-run-multus-certs\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.178099 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.178021 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-socket-dir-parent\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.178099 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.178033 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-conf-dir\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.178099 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.178040 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d50c29d-9104-4dd5-9499-7dd94594666b-root\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:39.178099 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.178061 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d50c29d-9104-4dd5-9499-7dd94594666b-root\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:39.178099 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.178061 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-run-multus-certs\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.178099 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.178087 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-wtmp\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:39.178099 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.178106 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-multus-socket-dir-parent\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.178701 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.178121 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-var-lib-kubelet\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.178701 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.178160 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-var-lib-cni-multus\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.178701 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.178184 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7cbe8a51-0318-450b-adb2-4abfe1097883-hosts-file\") pod \"node-resolver-7vrj7\" (UID: \"7cbe8a51-0318-450b-adb2-4abfe1097883\") " pod="openshift-dns/node-resolver-7vrj7" Nov 04 00:58:39.178701 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.178189 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d50c29d-9104-4dd5-9499-7dd94594666b-node-exporter-wtmp\") pod \"node-exporter-vqnkh\" (UID: \"5d50c29d-9104-4dd5-9499-7dd94594666b\") " pod="openshift-monitoring/node-exporter-vqnkh" Nov 04 00:58:39.178701 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.178211 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/565a71a6-1ce7-4078-90fc-f2ccadaa0d17-host\") pod \"node-ca-v8q47\" (UID: \"565a71a6-1ce7-4078-90fc-f2ccadaa0d17\") " pod="openshift-image-registry/node-ca-v8q47" Nov 04 00:58:39.178701 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.178225 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-var-lib-kubelet\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.178701 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.178237 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-var-lib-cni-multus\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.178701 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.178252 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7cbe8a51-0318-450b-adb2-4abfe1097883-hosts-file\") pod \"node-resolver-7vrj7\" (UID: \"7cbe8a51-0318-450b-adb2-4abfe1097883\") " pod="openshift-dns/node-resolver-7vrj7" Nov 04 00:58:39.178701 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.178258 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/565a71a6-1ce7-4078-90fc-f2ccadaa0d17-host\") pod \"node-ca-v8q47\" (UID: \"565a71a6-1ce7-4078-90fc-f2ccadaa0d17\") " pod="openshift-image-registry/node-ca-v8q47" Nov 04 00:58:39.178701 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.178273 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-var-lib-cni-bin\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.178701 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.178293 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-hostroot\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.178701 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.178305 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-host-var-lib-cni-bin\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:39.178701 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:39.178346 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac5e9e6b-7405-4c03-a149-4bf353209f24-hostroot\") pod \"multus-dxbxk\" (UID: \"ac5e9e6b-7405-4c03-a149-4bf353209f24\") " pod="openshift-multus/multus-dxbxk" Nov 04 00:58:41.342536 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:41.342505 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:41.344087 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:41.344064 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z9szb" Nov 04 00:58:41.919782 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:41.919731 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:42.047494 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:42.047464 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:44.885764 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:44.885731 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:58:45.411712 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:45.411686 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:48.374421 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:48.374384 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:58:48.374933 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:48.374606 4852 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:58:48.844918 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:48.844886 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:48.978581 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:48.978545 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:51.355902 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.355870 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 00:58:51.359500 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.359481 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.363158 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.363131 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 04 00:58:51.363373 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.363348 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 04 00:58:51.363373 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.363348 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 04 00:58:51.363534 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.363355 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 04 00:58:51.364391 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.364375 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 04 00:58:51.364468 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.364437 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 04 00:58:51.364663 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.364649 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 04 00:58:51.364733 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.364687 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-llw9w\"" Nov 04 00:58:51.371131 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.371101 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 04 00:58:51.372000 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.371969 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 04 00:58:51.373229 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.373206 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 00:58:51.436978 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.436952 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-web-config\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.437108 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.436987 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.437108 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.437003 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.437108 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.437025 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.437208 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.437126 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.437208 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.437174 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.437271 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.437221 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-config-volume\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.437271 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.437240 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.437339 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.437267 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.437339 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.437322 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.437400 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.437340 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-config-out\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.437400 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.437355 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.437400 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.437369 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ddjf\" (UniqueName: \"kubernetes.io/projected/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-kube-api-access-9ddjf\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.538449 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.538403 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.538542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.538455 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-config-out\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.538542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.538473 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.538542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.538490 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ddjf\" (UniqueName: \"kubernetes.io/projected/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-kube-api-access-9ddjf\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.538542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.538515 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-web-config\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.538542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.538537 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.538797 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.538551 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.538797 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.538586 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.538797 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.538613 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.538797 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.538637 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.538797 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.538666 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-config-volume\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.538797 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.538691 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.538797 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.538726 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.539126 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.538856 4852 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 00:58:51.539202 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.539176 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.539643 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.539612 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.539855 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.539826 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.542054 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.542029 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-web-config\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.542170 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.542150 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-config-out\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.542281 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.542231 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-config-volume\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.542380 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.542349 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.542605 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.542580 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.542693 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.542676 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.542764 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.542743 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.542813 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.542769 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.543865 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.543847 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.546364 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.546348 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ddjf\" (UniqueName: \"kubernetes.io/projected/dba769a9-3d04-4de1-b3c2-022fd9d98e7e-kube-api-access-9ddjf\") pod \"alertmanager-main-0\" (UID: \"dba769a9-3d04-4de1-b3c2-022fd9d98e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.677078 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.677024 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.956896 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.956839 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 00:58:51.958950 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:51.958917 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba769a9_3d04_4de1_b3c2_022fd9d98e7e.slice/crio-696423f16186b26ebe854f61d0208ba09daa4408cbfdda4ed8acfc4935b6a8b6 WatchSource:0}: Error finding container 696423f16186b26ebe854f61d0208ba09daa4408cbfdda4ed8acfc4935b6a8b6: Status 404 returned error can't find the container with id 696423f16186b26ebe854f61d0208ba09daa4408cbfdda4ed8acfc4935b6a8b6 Nov 04 00:58:51.961001 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:51.960983 4852 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 00:58:52.073895 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:52.073868 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dba769a9-3d04-4de1-b3c2-022fd9d98e7e","Type":"ContainerStarted","Data":"696423f16186b26ebe854f61d0208ba09daa4408cbfdda4ed8acfc4935b6a8b6"} Nov 04 00:58:53.664497 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.664460 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-8685b5f79f-x2l7l"] Nov 04 00:58:53.666647 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.666626 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.671627 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.671608 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 04 00:58:53.674924 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.674905 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 04 00:58:53.680874 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.680855 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8685b5f79f-x2l7l"] Nov 04 00:58:53.757829 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.757799 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/251d4dec-f4d2-4c9d-8734-eab92112714c-federate-client-tls\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.757963 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.757838 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/251d4dec-f4d2-4c9d-8734-eab92112714c-serving-certs-ca-bundle\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.757963 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.757874 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/251d4dec-f4d2-4c9d-8734-eab92112714c-telemeter-client-tls\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.758046 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.757966 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/251d4dec-f4d2-4c9d-8734-eab92112714c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.758046 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.758000 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2fs2\" (UniqueName: \"kubernetes.io/projected/251d4dec-f4d2-4c9d-8734-eab92112714c-kube-api-access-j2fs2\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.758046 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.758022 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/251d4dec-f4d2-4c9d-8734-eab92112714c-metrics-client-ca\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.758160 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.758123 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/251d4dec-f4d2-4c9d-8734-eab92112714c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.758160 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.758154 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/251d4dec-f4d2-4c9d-8734-eab92112714c-secret-telemeter-client\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.763014 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.762988 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 04 00:58:53.763136 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.763122 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 04 00:58:53.763185 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.763131 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 04 00:58:53.763185 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.763123 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 04 00:58:53.763254 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.763193 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-c5m5c\"" Nov 04 00:58:53.858752 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.858723 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/251d4dec-f4d2-4c9d-8734-eab92112714c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.858865 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.858763 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/251d4dec-f4d2-4c9d-8734-eab92112714c-secret-telemeter-client\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.858865 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.858796 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/251d4dec-f4d2-4c9d-8734-eab92112714c-federate-client-tls\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.858865 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.858825 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/251d4dec-f4d2-4c9d-8734-eab92112714c-serving-certs-ca-bundle\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.858865 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.858855 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/251d4dec-f4d2-4c9d-8734-eab92112714c-telemeter-client-tls\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.859046 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.858886 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/251d4dec-f4d2-4c9d-8734-eab92112714c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.859046 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.859023 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2fs2\" (UniqueName: \"kubernetes.io/projected/251d4dec-f4d2-4c9d-8734-eab92112714c-kube-api-access-j2fs2\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.859149 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.859069 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/251d4dec-f4d2-4c9d-8734-eab92112714c-metrics-client-ca\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.859931 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.859910 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/251d4dec-f4d2-4c9d-8734-eab92112714c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.860048 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.859957 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/251d4dec-f4d2-4c9d-8734-eab92112714c-serving-certs-ca-bundle\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.860048 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.860009 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/251d4dec-f4d2-4c9d-8734-eab92112714c-metrics-client-ca\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.861540 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.861520 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/251d4dec-f4d2-4c9d-8734-eab92112714c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.861708 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.861691 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/251d4dec-f4d2-4c9d-8734-eab92112714c-federate-client-tls\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.861971 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.861950 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/251d4dec-f4d2-4c9d-8734-eab92112714c-telemeter-client-tls\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.862006 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.861955 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/251d4dec-f4d2-4c9d-8734-eab92112714c-secret-telemeter-client\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.868725 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.868708 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2fs2\" (UniqueName: \"kubernetes.io/projected/251d4dec-f4d2-4c9d-8734-eab92112714c-kube-api-access-j2fs2\") pod \"telemeter-client-8685b5f79f-x2l7l\" (UID: \"251d4dec-f4d2-4c9d-8734-eab92112714c\") " pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:53.977823 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:53.977748 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" Nov 04 00:58:54.081542 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:54.081512 4852 generic.go:356] "Generic (PLEG): container finished" podID="dba769a9-3d04-4de1-b3c2-022fd9d98e7e" containerID="bba8b78e60e00761c438636c5e19fe455221f94f77ab54a3818020bed6f78826" exitCode=0 Nov 04 00:58:54.081679 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:54.081552 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dba769a9-3d04-4de1-b3c2-022fd9d98e7e","Type":"ContainerDied","Data":"bba8b78e60e00761c438636c5e19fe455221f94f77ab54a3818020bed6f78826"} Nov 04 00:58:54.248947 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:54.248868 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8685b5f79f-x2l7l"] Nov 04 00:58:54.251809 ip-10-0-140-241 kubenswrapper[4852]: W1104 00:58:54.251771 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251d4dec_f4d2_4c9d_8734_eab92112714c.slice/crio-0f86062fb8b2e199ba84e52a579d3b6384d559d0af25a0d3a9b45a0e60864352 WatchSource:0}: Error finding container 0f86062fb8b2e199ba84e52a579d3b6384d559d0af25a0d3a9b45a0e60864352: Status 404 returned error can't find the container with id 0f86062fb8b2e199ba84e52a579d3b6384d559d0af25a0d3a9b45a0e60864352 Nov 04 00:58:55.085697 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:55.085655 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" event={"ID":"251d4dec-f4d2-4c9d-8734-eab92112714c","Type":"ContainerStarted","Data":"0f86062fb8b2e199ba84e52a579d3b6384d559d0af25a0d3a9b45a0e60864352"} Nov 04 00:58:56.093507 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:56.093478 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dba769a9-3d04-4de1-b3c2-022fd9d98e7e","Type":"ContainerStarted","Data":"2af053268b7169d8d87f1711fdaf67b5b8bea84f6b8553262e8e69b5d40b0e1d"} Nov 04 00:58:57.004497 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:57.004465 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:57.100906 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:57.100872 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dba769a9-3d04-4de1-b3c2-022fd9d98e7e","Type":"ContainerStarted","Data":"4d732828359e467e83a28ac371e99449ba195208391869a3ef7e80c39ac004ad"} Nov 04 00:58:57.101354 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:57.100959 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dba769a9-3d04-4de1-b3c2-022fd9d98e7e","Type":"ContainerStarted","Data":"26958fd92dfb11b71ebdc32afa4d9bf4b950dc79ef0120a150086a738934d14e"} Nov 04 00:58:57.101354 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:57.101004 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dba769a9-3d04-4de1-b3c2-022fd9d98e7e","Type":"ContainerStarted","Data":"ed304349a046a21e37e2aefc3fe2fd01c30c26846197233e5d4428924099a850"} Nov 04 00:58:57.101354 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:57.101043 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dba769a9-3d04-4de1-b3c2-022fd9d98e7e","Type":"ContainerStarted","Data":"2cacf71f87f51d30c309485f4c08302f7fdc0256753e16e300cb17134c1f652d"} Nov 04 00:58:57.103274 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:57.103244 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" event={"ID":"251d4dec-f4d2-4c9d-8734-eab92112714c","Type":"ContainerStarted","Data":"014d4c3fa9d44fc0815bff3e8c7fa3a3d2db7bd5f0d5b0545c9b2ed48901b379"} Nov 04 00:58:57.103409 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:57.103295 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" event={"ID":"251d4dec-f4d2-4c9d-8734-eab92112714c","Type":"ContainerStarted","Data":"8510dfa6cbcc10686d0d1bedcf5255af817bde1c87c06e0acc08da79a46c1614"} Nov 04 00:58:57.103409 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:57.103325 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" event={"ID":"251d4dec-f4d2-4c9d-8734-eab92112714c","Type":"ContainerStarted","Data":"d4852b79aad3ddea84c7313ec65c7b442d6a3a00c4c4c2d0a035dc149fea1492"} Nov 04 00:58:57.126778 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:57.126712 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-8685b5f79f-x2l7l" podStartSLOduration=1.9713369809999999 podStartE2EDuration="4.126698133s" totalImagesPullingTime="2.155361152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:57.124930462 +0000 UTC m=+19.240040580" watchObservedRunningTime="2026-11-04 00:58:57.126698133 +0000 UTC m=+19.241808251" Nov 04 00:58:57.136501 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:57.136471 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:57.374802 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:57.374776 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:57.539731 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:57.539705 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:58.043988 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:58.043959 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:58.109610 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:58.109557 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dba769a9-3d04-4de1-b3c2-022fd9d98e7e","Type":"ContainerStarted","Data":"2cf2293f7dbf36e33090ef66924b8cd86812b764f30b7e158b26b8a7e3d9b388"} Nov 04 00:58:58.139457 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:58.139379 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.040491407 podStartE2EDuration="7.139367057s" totalImagesPullingTime="4.087285207s" totalInitContainerRuntime="11.590443ms" isStatefulPod=false podCreationTimestamp="2026-11-04 00:58:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:58.13782508 +0000 UTC m=+20.252935197" watchObservedRunningTime="2026-11-04 00:58:58.139367057 +0000 UTC m=+20.254477165" Nov 04 00:58:58.263532 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:58.263500 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:59.638626 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:58:59.638585 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8d5qr" Nov 04 00:58:59.864641 ip-10-0-140-241 kubenswrapper[4852]: E1104 00:58:59.864609 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 00:59:09.243969 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:59:09.243938 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vst89" Nov 04 00:59:14.888134 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:59:14.888062 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b4k7w" Nov 04 00:59:17.925070 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:59:17.925036 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:59:29.664517 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:59:29.664481 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:59:29.864578 ip-10-0-140-241 kubenswrapper[4852]: E1104 00:59:29.864531 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 00:59:33.900833 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:59:33.900797 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:59:34.038452 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:59:34.038417 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:59:34.297080 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:59:34.297050 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:59:34.433494 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:59:34.433467 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:59:34.714386 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:59:34.714317 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:59:34.847035 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:59:34.847009 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:59:46.529740 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:59:46.529700 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:59:46.661226 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:59:46.661179 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:59:54.209456 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:59:54.209415 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:59:56.904230 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:59:56.904194 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:59:57.033096 ip-10-0-140-241 kubenswrapper[4852]: I1104 00:59:57.033067 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 00:59:59.866244 ip-10-0-140-241 kubenswrapper[4852]: E1104 00:59:59.866210 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:00:29.713985 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:00:29.713957 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:00:29.864254 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:00:29.864217 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:00:59.865492 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:00:59.865405 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:01:29.779192 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:01:29.779156 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:01:29.865586 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:01:29.865528 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:01:59.865858 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:01:59.865815 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:02:02.634222 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:02.634191 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:02.810100 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:02.810061 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:03.025754 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.025722 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:03.582168 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.582138 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5"] Nov 04 01:02:03.585417 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.585397 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5" Nov 04 01:02:03.590420 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.590393 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 04 01:02:03.590420 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.590405 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 04 01:02:03.590661 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.590473 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-bhfmq\"" Nov 04 01:02:03.605440 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.605414 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5"] Nov 04 01:02:03.655794 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.655758 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22"] Nov 04 01:02:03.658974 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.658957 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22" Nov 04 01:02:03.667708 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.667684 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22"] Nov 04 01:02:03.689674 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.689649 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-9xfvx\"" Nov 04 01:02:03.711605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.711541 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjd2h\" (UniqueName: \"kubernetes.io/projected/98d7c48d-9bb2-48b2-9a70-5fea49fc2a93-kube-api-access-jjd2h\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5\" (UID: \"98d7c48d-9bb2-48b2-9a70-5fea49fc2a93\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5" Nov 04 01:02:03.711605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.711609 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98d7c48d-9bb2-48b2-9a70-5fea49fc2a93-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5\" (UID: \"98d7c48d-9bb2-48b2-9a70-5fea49fc2a93\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5" Nov 04 01:02:03.711801 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.711661 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98d7c48d-9bb2-48b2-9a70-5fea49fc2a93-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5\" (UID: \"98d7c48d-9bb2-48b2-9a70-5fea49fc2a93\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5" Nov 04 01:02:03.812389 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.812359 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa8a3b04-4df1-43fc-a750-04b464beb85b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22\" (UID: \"fa8a3b04-4df1-43fc-a750-04b464beb85b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22" Nov 04 01:02:03.812549 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.812406 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa8a3b04-4df1-43fc-a750-04b464beb85b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22\" (UID: \"fa8a3b04-4df1-43fc-a750-04b464beb85b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22" Nov 04 01:02:03.812549 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.812475 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjd2h\" (UniqueName: \"kubernetes.io/projected/98d7c48d-9bb2-48b2-9a70-5fea49fc2a93-kube-api-access-jjd2h\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5\" (UID: \"98d7c48d-9bb2-48b2-9a70-5fea49fc2a93\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5" Nov 04 01:02:03.812549 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.812507 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98d7c48d-9bb2-48b2-9a70-5fea49fc2a93-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5\" (UID: \"98d7c48d-9bb2-48b2-9a70-5fea49fc2a93\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5" Nov 04 01:02:03.812549 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.812523 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98d7c48d-9bb2-48b2-9a70-5fea49fc2a93-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5\" (UID: \"98d7c48d-9bb2-48b2-9a70-5fea49fc2a93\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5" Nov 04 01:02:03.812549 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.812544 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzqv9\" (UniqueName: \"kubernetes.io/projected/fa8a3b04-4df1-43fc-a750-04b464beb85b-kube-api-access-pzqv9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22\" (UID: \"fa8a3b04-4df1-43fc-a750-04b464beb85b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22" Nov 04 01:02:03.812956 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.812933 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98d7c48d-9bb2-48b2-9a70-5fea49fc2a93-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5\" (UID: \"98d7c48d-9bb2-48b2-9a70-5fea49fc2a93\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5" Nov 04 01:02:03.812996 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.812941 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98d7c48d-9bb2-48b2-9a70-5fea49fc2a93-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5\" (UID: \"98d7c48d-9bb2-48b2-9a70-5fea49fc2a93\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5" Nov 04 01:02:03.820966 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.820939 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjd2h\" (UniqueName: \"kubernetes.io/projected/98d7c48d-9bb2-48b2-9a70-5fea49fc2a93-kube-api-access-jjd2h\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5\" (UID: \"98d7c48d-9bb2-48b2-9a70-5fea49fc2a93\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5" Nov 04 01:02:03.901633 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.901518 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5" Nov 04 01:02:03.913682 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.913654 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa8a3b04-4df1-43fc-a750-04b464beb85b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22\" (UID: \"fa8a3b04-4df1-43fc-a750-04b464beb85b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22" Nov 04 01:02:03.913767 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.913698 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa8a3b04-4df1-43fc-a750-04b464beb85b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22\" (UID: \"fa8a3b04-4df1-43fc-a750-04b464beb85b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22" Nov 04 01:02:03.913767 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.913730 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzqv9\" (UniqueName: \"kubernetes.io/projected/fa8a3b04-4df1-43fc-a750-04b464beb85b-kube-api-access-pzqv9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22\" (UID: \"fa8a3b04-4df1-43fc-a750-04b464beb85b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22" Nov 04 01:02:03.914127 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.914102 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa8a3b04-4df1-43fc-a750-04b464beb85b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22\" (UID: \"fa8a3b04-4df1-43fc-a750-04b464beb85b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22" Nov 04 01:02:03.914206 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.914112 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa8a3b04-4df1-43fc-a750-04b464beb85b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22\" (UID: \"fa8a3b04-4df1-43fc-a750-04b464beb85b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22" Nov 04 01:02:03.926625 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.926583 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzqv9\" (UniqueName: \"kubernetes.io/projected/fa8a3b04-4df1-43fc-a750-04b464beb85b-kube-api-access-pzqv9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22\" (UID: \"fa8a3b04-4df1-43fc-a750-04b464beb85b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22" Nov 04 01:02:03.974998 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.974963 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22" Nov 04 01:02:03.994010 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.993985 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w"] Nov 04 01:02:03.997025 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:03.997006 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.001763 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.001678 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 04 01:02:04.001763 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.001676 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 04 01:02:04.001763 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.001713 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 04 01:02:04.001763 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.001714 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 04 01:02:04.001763 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.001717 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 04 01:02:04.002064 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.001809 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kwn4w\"" Nov 04 01:02:04.002064 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.001896 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 04 01:02:04.011419 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.011384 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w"] Nov 04 01:02:04.062619 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.062546 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:04.116701 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.116666 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d95e9a2f-2434-44cc-91a0-4841f4470299-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-l6g9w\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.116701 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.116701 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j749r\" (UniqueName: \"kubernetes.io/projected/d95e9a2f-2434-44cc-91a0-4841f4470299-kube-api-access-j749r\") pod \"istiod-openshift-gateway-7876bfcb74-l6g9w\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.116905 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.116734 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-l6g9w\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.116905 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.116862 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-l6g9w\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.116978 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.116925 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-l6g9w\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.116978 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.116955 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-l6g9w\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.116978 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.116972 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d95e9a2f-2434-44cc-91a0-4841f4470299-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-l6g9w\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.178045 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.177967 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5"] Nov 04 01:02:04.181141 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:02:04.181115 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d7c48d_9bb2_48b2_9a70_5fea49fc2a93.slice/crio-6ab6983f982dbdfed9b05b34cd7591be7391da1068229df76e142ebe96059c5b WatchSource:0}: Error finding container 6ab6983f982dbdfed9b05b34cd7591be7391da1068229df76e142ebe96059c5b: Status 404 returned error can't find the container with id 6ab6983f982dbdfed9b05b34cd7591be7391da1068229df76e142ebe96059c5b Nov 04 01:02:04.217757 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.217726 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-l6g9w\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.217757 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.217762 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d95e9a2f-2434-44cc-91a0-4841f4470299-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-l6g9w\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.217966 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.217788 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d95e9a2f-2434-44cc-91a0-4841f4470299-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-l6g9w\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.217966 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.217913 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j749r\" (UniqueName: \"kubernetes.io/projected/d95e9a2f-2434-44cc-91a0-4841f4470299-kube-api-access-j749r\") pod \"istiod-openshift-gateway-7876bfcb74-l6g9w\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.218047 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.217965 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-l6g9w\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.218047 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.218007 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-l6g9w\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.218147 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.218086 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-l6g9w\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.218682 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.218648 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-l6g9w\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.219866 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.219834 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:04.220348 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.220330 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d95e9a2f-2434-44cc-91a0-4841f4470299-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-l6g9w\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.220426 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.220355 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d95e9a2f-2434-44cc-91a0-4841f4470299-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-l6g9w\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.221060 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.221040 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-l6g9w\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.221133 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.221116 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-l6g9w\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.225725 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.225703 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j749r\" (UniqueName: \"kubernetes.io/projected/d95e9a2f-2434-44cc-91a0-4841f4470299-kube-api-access-j749r\") pod \"istiod-openshift-gateway-7876bfcb74-l6g9w\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.225826 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.225708 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-l6g9w\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.253773 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.253738 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22"] Nov 04 01:02:04.256903 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:02:04.256872 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8a3b04_4df1_43fc_a750_04b464beb85b.slice/crio-81532d40cd2cd6f3a423541bac9c0228d97293ae555d089280e1c5a94432d887 WatchSource:0}: Error finding container 81532d40cd2cd6f3a423541bac9c0228d97293ae555d089280e1c5a94432d887: Status 404 returned error can't find the container with id 81532d40cd2cd6f3a423541bac9c0228d97293ae555d089280e1c5a94432d887 Nov 04 01:02:04.316316 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.316276 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:04.587924 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.587888 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w"] Nov 04 01:02:04.591359 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:02:04.591322 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95e9a2f_2434_44cc_91a0_4841f4470299.slice/crio-514f4faf9ed025bf8bacad340ce5a9c5c24c34e6d150c1dd41f66eff39ce1999 WatchSource:0}: Error finding container 514f4faf9ed025bf8bacad340ce5a9c5c24c34e6d150c1dd41f66eff39ce1999: Status 404 returned error can't find the container with id 514f4faf9ed025bf8bacad340ce5a9c5c24c34e6d150c1dd41f66eff39ce1999 Nov 04 01:02:04.691510 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.691466 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22" event={"ID":"fa8a3b04-4df1-43fc-a750-04b464beb85b","Type":"ContainerStarted","Data":"81532d40cd2cd6f3a423541bac9c0228d97293ae555d089280e1c5a94432d887"} Nov 04 01:02:04.692659 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.692632 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" event={"ID":"d95e9a2f-2434-44cc-91a0-4841f4470299","Type":"ContainerStarted","Data":"514f4faf9ed025bf8bacad340ce5a9c5c24c34e6d150c1dd41f66eff39ce1999"} Nov 04 01:02:04.693742 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.693720 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5" event={"ID":"98d7c48d-9bb2-48b2-9a70-5fea49fc2a93","Type":"ContainerStarted","Data":"6ab6983f982dbdfed9b05b34cd7591be7391da1068229df76e142ebe96059c5b"} Nov 04 01:02:04.975337 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:04.975245 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:06.670184 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:06.670080 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:06.803400 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:06.803364 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:08.353444 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:08.353405 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:02:08.353875 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:08.353484 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:02:08.712487 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:08.712362 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" event={"ID":"d95e9a2f-2434-44cc-91a0-4841f4470299","Type":"ContainerStarted","Data":"9af1aaceebdcdf7573575bad99b46ae6ec3db0326b7cef94e540e002861047f2"} Nov 04 01:02:08.712669 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:08.712517 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:08.733405 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:08.733354 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" podStartSLOduration=1.974216838 podStartE2EDuration="5.733339764s" totalImagesPullingTime="3.759122926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:08.732116014 +0000 UTC m=+210.847226135" watchObservedRunningTime="2026-11-04 01:02:08.733339764 +0000 UTC m=+210.848449882" Nov 04 01:02:09.004353 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:09.004319 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:09.137371 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:09.137342 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:09.718290 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:09.718244 4852 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-l6g9w container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:02:09.718935 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:09.718313 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" podUID="d95e9a2f-2434-44cc-91a0-4841f4470299" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:02:09.808466 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:09.808419 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:09.948126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:09.948081 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:10.084771 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:10.084708 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:12.704230 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:12.704147 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:12.718010 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:12.717977 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:02:13.733080 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:13.733044 4852 generic.go:356] "Generic (PLEG): container finished" podID="98d7c48d-9bb2-48b2-9a70-5fea49fc2a93" containerID="4c78fc77fb2663159c3f1d2690021943bdd3ac970b68bdf74062a6c2618d0716" exitCode=0 Nov 04 01:02:13.733545 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:13.733133 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5" event={"ID":"98d7c48d-9bb2-48b2-9a70-5fea49fc2a93","Type":"ContainerDied","Data":"4c78fc77fb2663159c3f1d2690021943bdd3ac970b68bdf74062a6c2618d0716"} Nov 04 01:02:13.734702 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:13.734679 4852 generic.go:356] "Generic (PLEG): container finished" podID="fa8a3b04-4df1-43fc-a750-04b464beb85b" containerID="b3c4959844037a8130923289d288ba64d5cb8e21f9ff3c4ac3c89e6523e3cac3" exitCode=0 Nov 04 01:02:13.734796 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:13.734710 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22" event={"ID":"fa8a3b04-4df1-43fc-a750-04b464beb85b","Type":"ContainerDied","Data":"b3c4959844037a8130923289d288ba64d5cb8e21f9ff3c4ac3c89e6523e3cac3"} Nov 04 01:02:16.748456 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:16.748414 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22" event={"ID":"fa8a3b04-4df1-43fc-a750-04b464beb85b","Type":"ContainerStarted","Data":"671f57c58e5623e87e4744a0dd8900973af605ab4505bae09cd83cdcfe39e7b3"} Nov 04 01:02:17.754477 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:17.754445 4852 generic.go:356] "Generic (PLEG): container finished" podID="fa8a3b04-4df1-43fc-a750-04b464beb85b" containerID="671f57c58e5623e87e4744a0dd8900973af605ab4505bae09cd83cdcfe39e7b3" exitCode=0 Nov 04 01:02:17.754937 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:17.754585 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22" event={"ID":"fa8a3b04-4df1-43fc-a750-04b464beb85b","Type":"ContainerDied","Data":"671f57c58e5623e87e4744a0dd8900973af605ab4505bae09cd83cdcfe39e7b3"} Nov 04 01:02:18.265925 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:18.265886 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:19.764883 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:19.764849 4852 generic.go:356] "Generic (PLEG): container finished" podID="98d7c48d-9bb2-48b2-9a70-5fea49fc2a93" containerID="a2ef794223b915a91f25e9b2129bbdfe7d8b26a344dc80acc31a205c99507a96" exitCode=0 Nov 04 01:02:19.765384 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:19.764919 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5" event={"ID":"98d7c48d-9bb2-48b2-9a70-5fea49fc2a93","Type":"ContainerDied","Data":"a2ef794223b915a91f25e9b2129bbdfe7d8b26a344dc80acc31a205c99507a96"} Nov 04 01:02:26.793252 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:26.793216 4852 generic.go:356] "Generic (PLEG): container finished" podID="fa8a3b04-4df1-43fc-a750-04b464beb85b" containerID="fb0869347788c3fea547a39754b4cd3604ba45adbea3a905f31e52c4d0365dde" exitCode=0 Nov 04 01:02:26.793769 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:26.793289 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22" event={"ID":"fa8a3b04-4df1-43fc-a750-04b464beb85b","Type":"ContainerDied","Data":"fb0869347788c3fea547a39754b4cd3604ba45adbea3a905f31e52c4d0365dde"} Nov 04 01:02:26.795396 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:26.795371 4852 generic.go:356] "Generic (PLEG): container finished" podID="98d7c48d-9bb2-48b2-9a70-5fea49fc2a93" containerID="beabf3abd1cc60ad04107c4be315b5f68c7cc8d23d3b4c49fc9d0b6ba9d39e8e" exitCode=0 Nov 04 01:02:26.795522 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:26.795445 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5" event={"ID":"98d7c48d-9bb2-48b2-9a70-5fea49fc2a93","Type":"ContainerDied","Data":"beabf3abd1cc60ad04107c4be315b5f68c7cc8d23d3b4c49fc9d0b6ba9d39e8e"} Nov 04 01:02:27.961531 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:27.961497 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5" Nov 04 01:02:27.977397 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:27.977365 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22" Nov 04 01:02:28.019305 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.019274 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8a3b04-4df1-43fc-a750-04b464beb85b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa8a3b04-4df1-43fc-a750-04b464beb85b-bundle\") pod \"fa8a3b04-4df1-43fc-a750-04b464beb85b\" (UID: \"fa8a3b04-4df1-43fc-a750-04b464beb85b\") " Nov 04 01:02:28.019505 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.019311 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8a3b04-4df1-43fc-a750-04b464beb85b-util\" (UniqueName: \"kubernetes.io/empty-dir/fa8a3b04-4df1-43fc-a750-04b464beb85b-util\") pod \"fa8a3b04-4df1-43fc-a750-04b464beb85b\" (UID: \"fa8a3b04-4df1-43fc-a750-04b464beb85b\") " Nov 04 01:02:28.019505 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.019366 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d7c48d-9bb2-48b2-9a70-5fea49fc2a93-util\" (UniqueName: \"kubernetes.io/empty-dir/98d7c48d-9bb2-48b2-9a70-5fea49fc2a93-util\") pod \"98d7c48d-9bb2-48b2-9a70-5fea49fc2a93\" (UID: \"98d7c48d-9bb2-48b2-9a70-5fea49fc2a93\") " Nov 04 01:02:28.019505 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.019399 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa8a3b04-4df1-43fc-a750-04b464beb85b-kube-api-access-pzqv9\" (UniqueName: \"kubernetes.io/projected/fa8a3b04-4df1-43fc-a750-04b464beb85b-kube-api-access-pzqv9\") pod \"fa8a3b04-4df1-43fc-a750-04b464beb85b\" (UID: \"fa8a3b04-4df1-43fc-a750-04b464beb85b\") " Nov 04 01:02:28.019505 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.019444 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98d7c48d-9bb2-48b2-9a70-5fea49fc2a93-kube-api-access-jjd2h\" (UniqueName: \"kubernetes.io/projected/98d7c48d-9bb2-48b2-9a70-5fea49fc2a93-kube-api-access-jjd2h\") pod \"98d7c48d-9bb2-48b2-9a70-5fea49fc2a93\" (UID: \"98d7c48d-9bb2-48b2-9a70-5fea49fc2a93\") " Nov 04 01:02:28.019505 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.019478 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d7c48d-9bb2-48b2-9a70-5fea49fc2a93-bundle\" (UniqueName: \"kubernetes.io/empty-dir/98d7c48d-9bb2-48b2-9a70-5fea49fc2a93-bundle\") pod \"98d7c48d-9bb2-48b2-9a70-5fea49fc2a93\" (UID: \"98d7c48d-9bb2-48b2-9a70-5fea49fc2a93\") " Nov 04 01:02:28.020066 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.020037 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d7c48d-9bb2-48b2-9a70-5fea49fc2a93-bundle" pod "98d7c48d-9bb2-48b2-9a70-5fea49fc2a93" (UID: "98d7c48d-9bb2-48b2-9a70-5fea49fc2a93"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:02:28.020308 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.020283 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8a3b04-4df1-43fc-a750-04b464beb85b-bundle" pod "fa8a3b04-4df1-43fc-a750-04b464beb85b" (UID: "fa8a3b04-4df1-43fc-a750-04b464beb85b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:02:28.021973 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.021950 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8a3b04-4df1-43fc-a750-04b464beb85b-kube-api-access-pzqv9" pod "fa8a3b04-4df1-43fc-a750-04b464beb85b" (UID: "fa8a3b04-4df1-43fc-a750-04b464beb85b"). InnerVolumeSpecName "kube-api-access-pzqv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:02:28.022154 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.022132 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d7c48d-9bb2-48b2-9a70-5fea49fc2a93-kube-api-access-jjd2h" pod "98d7c48d-9bb2-48b2-9a70-5fea49fc2a93" (UID: "98d7c48d-9bb2-48b2-9a70-5fea49fc2a93"). InnerVolumeSpecName "kube-api-access-jjd2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:02:28.025875 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.025846 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d7c48d-9bb2-48b2-9a70-5fea49fc2a93-util" pod "98d7c48d-9bb2-48b2-9a70-5fea49fc2a93" (UID: "98d7c48d-9bb2-48b2-9a70-5fea49fc2a93"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:02:28.026243 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.026223 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8a3b04-4df1-43fc-a750-04b464beb85b-util" pod "fa8a3b04-4df1-43fc-a750-04b464beb85b" (UID: "fa8a3b04-4df1-43fc-a750-04b464beb85b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:02:28.120093 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.119995 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjd2h\" (UniqueName: \"kubernetes.io/projected/98d7c48d-9bb2-48b2-9a70-5fea49fc2a93-kube-api-access-jjd2h\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:28.120093 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.120026 4852 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98d7c48d-9bb2-48b2-9a70-5fea49fc2a93-bundle\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:28.120093 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.120038 4852 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa8a3b04-4df1-43fc-a750-04b464beb85b-bundle\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:28.120093 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.120048 4852 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa8a3b04-4df1-43fc-a750-04b464beb85b-util\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:28.120093 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.120059 4852 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98d7c48d-9bb2-48b2-9a70-5fea49fc2a93-util\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:28.120093 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.120072 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzqv9\" (UniqueName: \"kubernetes.io/projected/fa8a3b04-4df1-43fc-a750-04b464beb85b-kube-api-access-pzqv9\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:28.804029 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.803996 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5" Nov 04 01:02:28.804029 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.804008 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449ccs5" event={"ID":"98d7c48d-9bb2-48b2-9a70-5fea49fc2a93","Type":"ContainerDied","Data":"6ab6983f982dbdfed9b05b34cd7591be7391da1068229df76e142ebe96059c5b"} Nov 04 01:02:28.804251 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.804091 4852 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ab6983f982dbdfed9b05b34cd7591be7391da1068229df76e142ebe96059c5b" Nov 04 01:02:28.805933 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.805907 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22" event={"ID":"fa8a3b04-4df1-43fc-a750-04b464beb85b","Type":"ContainerDied","Data":"81532d40cd2cd6f3a423541bac9c0228d97293ae555d089280e1c5a94432d887"} Nov 04 01:02:28.806049 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.805942 4852 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81532d40cd2cd6f3a423541bac9c0228d97293ae555d089280e1c5a94432d887" Nov 04 01:02:28.806049 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:28.805954 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c85w22" Nov 04 01:02:29.806832 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:29.806783 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:29.865231 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:02:29.865196 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:02:43.720859 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:43.720824 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:43.867111 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:43.867074 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:44.785627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:44.785592 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:44.939412 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:44.939377 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:46.646778 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:46.646742 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:46.781743 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:46.781708 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:50.131192 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:50.131158 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:56.757258 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:56.757230 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:56.964085 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:56.964057 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:57.974260 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:57.974210 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:58.080331 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.080298 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5"] Nov 04 01:02:58.080666 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.080649 4852 cpu_manager.go:414] "containerMap: removing container" podUID="98d7c48d-9bb2-48b2-9a70-5fea49fc2a93" containerName="pull" Nov 04 01:02:58.080731 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.080668 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98d7c48d-9bb2-48b2-9a70-5fea49fc2a93" containerName="pull" Nov 04 01:02:58.080731 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.080678 4852 cpu_manager.go:414] "containerMap: removing container" podUID="98d7c48d-9bb2-48b2-9a70-5fea49fc2a93" containerName="extract" Nov 04 01:02:58.080731 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.080683 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98d7c48d-9bb2-48b2-9a70-5fea49fc2a93" containerName="extract" Nov 04 01:02:58.080731 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.080698 4852 cpu_manager.go:414] "containerMap: removing container" podUID="fa8a3b04-4df1-43fc-a750-04b464beb85b" containerName="util" Nov 04 01:02:58.080731 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.080703 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa8a3b04-4df1-43fc-a750-04b464beb85b" containerName="util" Nov 04 01:02:58.080731 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.080709 4852 cpu_manager.go:414] "containerMap: removing container" podUID="fa8a3b04-4df1-43fc-a750-04b464beb85b" containerName="pull" Nov 04 01:02:58.080731 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.080713 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa8a3b04-4df1-43fc-a750-04b464beb85b" containerName="pull" Nov 04 01:02:58.080731 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.080723 4852 cpu_manager.go:414] "containerMap: removing container" podUID="fa8a3b04-4df1-43fc-a750-04b464beb85b" containerName="extract" Nov 04 01:02:58.080731 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.080729 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa8a3b04-4df1-43fc-a750-04b464beb85b" containerName="extract" Nov 04 01:02:58.081022 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.080742 4852 cpu_manager.go:414] "containerMap: removing container" podUID="98d7c48d-9bb2-48b2-9a70-5fea49fc2a93" containerName="util" Nov 04 01:02:58.081022 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.080749 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98d7c48d-9bb2-48b2-9a70-5fea49fc2a93" containerName="util" Nov 04 01:02:58.081022 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.080796 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="98d7c48d-9bb2-48b2-9a70-5fea49fc2a93" containerName="extract" Nov 04 01:02:58.081022 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.080806 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa8a3b04-4df1-43fc-a750-04b464beb85b" containerName="extract" Nov 04 01:02:58.098912 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.098883 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" Nov 04 01:02:58.113770 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.113737 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5"] Nov 04 01:02:58.138237 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.138201 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39fe0425-422d-4e5a-a269-04ccb4b6f169-tmp\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-bscj5\" (UID: \"39fe0425-422d-4e5a-a269-04ccb4b6f169\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" Nov 04 01:02:58.138237 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.138244 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4w65\" (UniqueName: \"kubernetes.io/projected/39fe0425-422d-4e5a-a269-04ccb4b6f169-kube-api-access-h4w65\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-bscj5\" (UID: \"39fe0425-422d-4e5a-a269-04ccb4b6f169\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" Nov 04 01:02:58.138471 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.138370 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/39fe0425-422d-4e5a-a269-04ccb4b6f169-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-bscj5\" (UID: \"39fe0425-422d-4e5a-a269-04ccb4b6f169\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" Nov 04 01:02:58.138471 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.138394 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/39fe0425-422d-4e5a-a269-04ccb4b6f169-manifests\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-bscj5\" (UID: \"39fe0425-422d-4e5a-a269-04ccb4b6f169\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" Nov 04 01:02:58.138471 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.138414 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39fe0425-422d-4e5a-a269-04ccb4b6f169-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-bscj5\" (UID: \"39fe0425-422d-4e5a-a269-04ccb4b6f169\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" Nov 04 01:02:58.138670 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.138471 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/39fe0425-422d-4e5a-a269-04ccb4b6f169-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-bscj5\" (UID: \"39fe0425-422d-4e5a-a269-04ccb4b6f169\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" Nov 04 01:02:58.154406 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.154378 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 04 01:02:58.154552 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.154415 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 04 01:02:58.154696 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.154673 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 04 01:02:58.154823 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.154709 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 04 01:02:58.154823 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.154783 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-9rtlr\"" Nov 04 01:02:58.154909 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.154827 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 04 01:02:58.239919 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.239831 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/39fe0425-422d-4e5a-a269-04ccb4b6f169-manifests\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-bscj5\" (UID: \"39fe0425-422d-4e5a-a269-04ccb4b6f169\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" Nov 04 01:02:58.239919 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.239868 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39fe0425-422d-4e5a-a269-04ccb4b6f169-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-bscj5\" (UID: \"39fe0425-422d-4e5a-a269-04ccb4b6f169\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" Nov 04 01:02:58.239919 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.239893 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/39fe0425-422d-4e5a-a269-04ccb4b6f169-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-bscj5\" (UID: \"39fe0425-422d-4e5a-a269-04ccb4b6f169\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" Nov 04 01:02:58.240208 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.239923 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39fe0425-422d-4e5a-a269-04ccb4b6f169-tmp\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-bscj5\" (UID: \"39fe0425-422d-4e5a-a269-04ccb4b6f169\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" Nov 04 01:02:58.240208 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.239956 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4w65\" (UniqueName: \"kubernetes.io/projected/39fe0425-422d-4e5a-a269-04ccb4b6f169-kube-api-access-h4w65\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-bscj5\" (UID: \"39fe0425-422d-4e5a-a269-04ccb4b6f169\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" Nov 04 01:02:58.240208 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.240109 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/39fe0425-422d-4e5a-a269-04ccb4b6f169-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-bscj5\" (UID: \"39fe0425-422d-4e5a-a269-04ccb4b6f169\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" Nov 04 01:02:58.240361 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.240307 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/39fe0425-422d-4e5a-a269-04ccb4b6f169-manifests\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-bscj5\" (UID: \"39fe0425-422d-4e5a-a269-04ccb4b6f169\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" Nov 04 01:02:58.242552 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.242527 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39fe0425-422d-4e5a-a269-04ccb4b6f169-tmp\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-bscj5\" (UID: \"39fe0425-422d-4e5a-a269-04ccb4b6f169\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" Nov 04 01:02:58.242700 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.242674 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/39fe0425-422d-4e5a-a269-04ccb4b6f169-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-bscj5\" (UID: \"39fe0425-422d-4e5a-a269-04ccb4b6f169\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" Nov 04 01:02:58.242751 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.242721 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39fe0425-422d-4e5a-a269-04ccb4b6f169-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-bscj5\" (UID: \"39fe0425-422d-4e5a-a269-04ccb4b6f169\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" Nov 04 01:02:58.242802 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.242787 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/39fe0425-422d-4e5a-a269-04ccb4b6f169-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-bscj5\" (UID: \"39fe0425-422d-4e5a-a269-04ccb4b6f169\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" Nov 04 01:02:58.247882 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.247857 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4w65\" (UniqueName: \"kubernetes.io/projected/39fe0425-422d-4e5a-a269-04ccb4b6f169-kube-api-access-h4w65\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-bscj5\" (UID: \"39fe0425-422d-4e5a-a269-04ccb4b6f169\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" Nov 04 01:02:58.295127 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.295086 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:58.437587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.437533 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" Nov 04 01:02:58.458851 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.458806 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:58.706072 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.706039 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:58.760776 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.760751 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5"] Nov 04 01:02:58.762409 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:02:58.762369 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39fe0425_422d_4e5a_a269_04ccb4b6f169.slice/crio-87ecbfd40936a2afa0e5244ae17a3e7b29be1c898a895b4c977743109eda8121 WatchSource:0}: Error finding container 87ecbfd40936a2afa0e5244ae17a3e7b29be1c898a895b4c977743109eda8121: Status 404 returned error can't find the container with id 87ecbfd40936a2afa0e5244ae17a3e7b29be1c898a895b4c977743109eda8121 Nov 04 01:02:58.905273 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:58.905238 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" event={"ID":"39fe0425-422d-4e5a-a269-04ccb4b6f169","Type":"ContainerStarted","Data":"87ecbfd40936a2afa0e5244ae17a3e7b29be1c898a895b4c977743109eda8121"} Nov 04 01:02:59.209075 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:59.208988 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:59.419752 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:02:59.419718 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:59.867468 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:02:59.867426 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:03:00.932730 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:00.932679 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:01.566756 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:01.566708 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:01.859654 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:01.859556 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:02.654190 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:02.654153 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-qww55"] Nov 04 01:03:02.664215 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:02.664190 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qww55" Nov 04 01:03:02.667428 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:02.667399 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-qww55"] Nov 04 01:03:02.670762 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:02.670667 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 04 01:03:02.670762 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:02.670718 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 04 01:03:02.670762 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:02.670726 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-mbjwb\"" Nov 04 01:03:02.777955 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:02.777914 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d8c8dd8-fc8a-48fb-83a2-b75fc5878ae3-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-qww55\" (UID: \"6d8c8dd8-fc8a-48fb-83a2-b75fc5878ae3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qww55" Nov 04 01:03:02.778158 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:02.778021 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg4zs\" (UniqueName: \"kubernetes.io/projected/6d8c8dd8-fc8a-48fb-83a2-b75fc5878ae3-kube-api-access-pg4zs\") pod \"cert-manager-cainjector-5c57bbd4fc-qww55\" (UID: \"6d8c8dd8-fc8a-48fb-83a2-b75fc5878ae3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qww55" Nov 04 01:03:02.878868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:02.878779 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg4zs\" (UniqueName: \"kubernetes.io/projected/6d8c8dd8-fc8a-48fb-83a2-b75fc5878ae3-kube-api-access-pg4zs\") pod \"cert-manager-cainjector-5c57bbd4fc-qww55\" (UID: \"6d8c8dd8-fc8a-48fb-83a2-b75fc5878ae3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qww55" Nov 04 01:03:02.878868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:02.878826 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d8c8dd8-fc8a-48fb-83a2-b75fc5878ae3-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-qww55\" (UID: \"6d8c8dd8-fc8a-48fb-83a2-b75fc5878ae3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qww55" Nov 04 01:03:02.887581 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:02.887528 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg4zs\" (UniqueName: \"kubernetes.io/projected/6d8c8dd8-fc8a-48fb-83a2-b75fc5878ae3-kube-api-access-pg4zs\") pod \"cert-manager-cainjector-5c57bbd4fc-qww55\" (UID: \"6d8c8dd8-fc8a-48fb-83a2-b75fc5878ae3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qww55" Nov 04 01:03:02.888060 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:02.888043 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d8c8dd8-fc8a-48fb-83a2-b75fc5878ae3-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-qww55\" (UID: \"6d8c8dd8-fc8a-48fb-83a2-b75fc5878ae3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qww55" Nov 04 01:03:02.983048 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:02.982963 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qww55" Nov 04 01:03:03.257777 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:03.257746 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-qww55"] Nov 04 01:03:03.261085 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:03:03.261056 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d8c8dd8_fc8a_48fb_83a2_b75fc5878ae3.slice/crio-8f33ad42c184a5a5cd84b19007bcd0227066e070bcffe05b1d11ba6c88943bd4 WatchSource:0}: Error finding container 8f33ad42c184a5a5cd84b19007bcd0227066e070bcffe05b1d11ba6c88943bd4: Status 404 returned error can't find the container with id 8f33ad42c184a5a5cd84b19007bcd0227066e070bcffe05b1d11ba6c88943bd4 Nov 04 01:03:03.925555 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:03.925518 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" event={"ID":"39fe0425-422d-4e5a-a269-04ccb4b6f169","Type":"ContainerStarted","Data":"b26f275c1b858d0c370e40cb11b0496f150f516105ccfcd133010ebee929f96c"} Nov 04 01:03:03.926792 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:03.926767 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qww55" event={"ID":"6d8c8dd8-fc8a-48fb-83a2-b75fc5878ae3","Type":"ContainerStarted","Data":"8f33ad42c184a5a5cd84b19007bcd0227066e070bcffe05b1d11ba6c88943bd4"} Nov 04 01:03:04.933448 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:04.933409 4852 generic.go:356] "Generic (PLEG): container finished" podID="39fe0425-422d-4e5a-a269-04ccb4b6f169" containerID="b26f275c1b858d0c370e40cb11b0496f150f516105ccfcd133010ebee929f96c" exitCode=0 Nov 04 01:03:04.933905 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:04.933495 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" event={"ID":"39fe0425-422d-4e5a-a269-04ccb4b6f169","Type":"ContainerDied","Data":"b26f275c1b858d0c370e40cb11b0496f150f516105ccfcd133010ebee929f96c"} Nov 04 01:03:05.183735 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:05.183651 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:05.939471 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:05.939430 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" event={"ID":"39fe0425-422d-4e5a-a269-04ccb4b6f169","Type":"ContainerStarted","Data":"da54b0dfec0c2259f0d5cfde4086a260a9dbf21743e8254fc4b218e744572494"} Nov 04 01:03:05.939880 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:05.939532 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" Nov 04 01:03:05.967323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:05.967286 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:05.992280 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:05.992233 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" podStartSLOduration=1.997204123 podStartE2EDuration="7.992218684s" totalImagesPullingTime="5.445640911s" totalInitContainerRuntime="549.37365ms" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:05.990462565 +0000 UTC m=+268.105572682" watchObservedRunningTime="2026-11-04 01:03:05.992218684 +0000 UTC m=+268.107328795" Nov 04 01:03:06.765717 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:06.765686 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:07.949413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:07.949319 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qww55" event={"ID":"6d8c8dd8-fc8a-48fb-83a2-b75fc5878ae3","Type":"ContainerStarted","Data":"9268add91ca3c4670c9c52172e9d566059935d9e396e9e6009d3bc7d2fd96d56"} Nov 04 01:03:07.983173 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:07.983119 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qww55" podStartSLOduration=1.633659104 podStartE2EDuration="5.983102765s" totalImagesPullingTime="4.349443661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:07.982082639 +0000 UTC m=+270.097192759" watchObservedRunningTime="2026-11-04 01:03:07.983102765 +0000 UTC m=+270.098212883" Nov 04 01:03:08.147423 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:08.147385 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:08.281065 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:08.281035 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:08.575629 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:08.575490 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:08.715236 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:08.715198 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:08.885355 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:08.885265 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:09.357336 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:09.357306 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:09.930044 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:09.930014 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:10.928974 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:10.928946 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:11.529357 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:11.529317 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:12.410653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:12.410614 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:12.589045 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:12.589010 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:13.457965 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:13.457931 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:13.647139 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:13.647108 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:13.886966 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:13.886936 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:14.055013 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:14.054982 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:16.758498 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:16.758461 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:16.911461 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:16.911428 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:16.946018 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:16.945986 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-bscj5" Nov 04 01:03:19.227524 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:19.227487 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-b9nwg"] Nov 04 01:03:19.230515 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:19.230497 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b9nwg" Nov 04 01:03:19.237762 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:19.237739 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 04 01:03:19.237915 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:19.237838 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-qmjlq\"" Nov 04 01:03:19.237915 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:19.237885 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 04 01:03:19.241788 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:19.241764 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-b9nwg"] Nov 04 01:03:19.312668 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:19.312627 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgmhg\" (UniqueName: \"kubernetes.io/projected/742db8d8-0dc9-43b5-a50d-45ab95c45bd3-kube-api-access-cgmhg\") pod \"limitador-operator-controller-manager-5778bb4449-b9nwg\" (UID: \"742db8d8-0dc9-43b5-a50d-45ab95c45bd3\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b9nwg" Nov 04 01:03:19.413696 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:19.413663 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgmhg\" (UniqueName: \"kubernetes.io/projected/742db8d8-0dc9-43b5-a50d-45ab95c45bd3-kube-api-access-cgmhg\") pod \"limitador-operator-controller-manager-5778bb4449-b9nwg\" (UID: \"742db8d8-0dc9-43b5-a50d-45ab95c45bd3\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b9nwg" Nov 04 01:03:19.435618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:19.435589 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgmhg\" (UniqueName: \"kubernetes.io/projected/742db8d8-0dc9-43b5-a50d-45ab95c45bd3-kube-api-access-cgmhg\") pod \"limitador-operator-controller-manager-5778bb4449-b9nwg\" (UID: \"742db8d8-0dc9-43b5-a50d-45ab95c45bd3\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b9nwg" Nov 04 01:03:19.542886 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:19.542836 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b9nwg" Nov 04 01:03:19.820416 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:19.820213 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-b9nwg"] Nov 04 01:03:19.823083 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:03:19.823055 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742db8d8_0dc9_43b5_a50d_45ab95c45bd3.slice/crio-0e977505ed5aa5c8d43e6965a6addf0d6adea6cddbd384d7294f17a99502bbf5 WatchSource:0}: Error finding container 0e977505ed5aa5c8d43e6965a6addf0d6adea6cddbd384d7294f17a99502bbf5: Status 404 returned error can't find the container with id 0e977505ed5aa5c8d43e6965a6addf0d6adea6cddbd384d7294f17a99502bbf5 Nov 04 01:03:19.994452 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:19.994418 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b9nwg" event={"ID":"742db8d8-0dc9-43b5-a50d-45ab95c45bd3","Type":"ContainerStarted","Data":"0e977505ed5aa5c8d43e6965a6addf0d6adea6cddbd384d7294f17a99502bbf5"} Nov 04 01:03:20.476458 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:20.476416 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:20.652512 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:20.652476 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:24.011386 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:24.011346 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b9nwg" event={"ID":"742db8d8-0dc9-43b5-a50d-45ab95c45bd3","Type":"ContainerStarted","Data":"e3a220f0d89a736178acfe2d08dea2bea9b2b6a04b933d8c47e00b3978517b24"} Nov 04 01:03:24.011903 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:24.011460 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b9nwg" Nov 04 01:03:24.028755 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:24.028703 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b9nwg" podStartSLOduration=1.5101224709999999 podStartE2EDuration="5.028689301s" totalImagesPullingTime="3.51856683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:24.027353031 +0000 UTC m=+286.142463151" watchObservedRunningTime="2026-11-04 01:03:24.028689301 +0000 UTC m=+286.143799418" Nov 04 01:03:24.945884 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:24.945840 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:25.095923 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:25.095879 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:25.547215 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:25.547180 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:25.698743 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:25.698709 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:25.963721 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:25.963633 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:26.168390 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:26.168358 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:29.852525 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:29.852479 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:29.865017 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:03:29.864981 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:03:30.118052 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:30.117968 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:30.272065 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:30.272034 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:31.903769 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:31.903734 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:32.067309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:32.067267 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:32.125849 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:32.125820 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-85448bfccc-jdm4z"] Nov 04 01:03:32.128915 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:32.128898 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85448bfccc-jdm4z" Nov 04 01:03:32.138010 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:32.137988 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85448bfccc-jdm4z"] Nov 04 01:03:32.204555 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:32.204394 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 04 01:03:32.204768 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:32.204552 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 04 01:03:32.204768 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:32.204668 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-47vck\"" Nov 04 01:03:32.253933 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:32.253893 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:32.325186 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:32.325146 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e5369da-d313-4e30-a3b4-cf31dfc43bbf-tmp\") pod \"kube-auth-proxy-85448bfccc-jdm4z\" (UID: \"3e5369da-d313-4e30-a3b4-cf31dfc43bbf\") " pod="openshift-ingress/kube-auth-proxy-85448bfccc-jdm4z" Nov 04 01:03:32.325357 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:32.325200 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e5369da-d313-4e30-a3b4-cf31dfc43bbf-tls-certs\") pod \"kube-auth-proxy-85448bfccc-jdm4z\" (UID: \"3e5369da-d313-4e30-a3b4-cf31dfc43bbf\") " pod="openshift-ingress/kube-auth-proxy-85448bfccc-jdm4z" Nov 04 01:03:32.325357 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:32.325270 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mvgt\" (UniqueName: \"kubernetes.io/projected/3e5369da-d313-4e30-a3b4-cf31dfc43bbf-kube-api-access-8mvgt\") pod \"kube-auth-proxy-85448bfccc-jdm4z\" (UID: \"3e5369da-d313-4e30-a3b4-cf31dfc43bbf\") " pod="openshift-ingress/kube-auth-proxy-85448bfccc-jdm4z" Nov 04 01:03:32.420349 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:32.420311 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:32.426424 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:32.426398 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e5369da-d313-4e30-a3b4-cf31dfc43bbf-tls-certs\") pod \"kube-auth-proxy-85448bfccc-jdm4z\" (UID: \"3e5369da-d313-4e30-a3b4-cf31dfc43bbf\") " pod="openshift-ingress/kube-auth-proxy-85448bfccc-jdm4z" Nov 04 01:03:32.426532 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:32.426438 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mvgt\" (UniqueName: \"kubernetes.io/projected/3e5369da-d313-4e30-a3b4-cf31dfc43bbf-kube-api-access-8mvgt\") pod \"kube-auth-proxy-85448bfccc-jdm4z\" (UID: \"3e5369da-d313-4e30-a3b4-cf31dfc43bbf\") " pod="openshift-ingress/kube-auth-proxy-85448bfccc-jdm4z" Nov 04 01:03:32.426532 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:32.426481 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e5369da-d313-4e30-a3b4-cf31dfc43bbf-tmp\") pod \"kube-auth-proxy-85448bfccc-jdm4z\" (UID: \"3e5369da-d313-4e30-a3b4-cf31dfc43bbf\") " pod="openshift-ingress/kube-auth-proxy-85448bfccc-jdm4z" Nov 04 01:03:32.428879 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:32.428851 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e5369da-d313-4e30-a3b4-cf31dfc43bbf-tmp\") pod \"kube-auth-proxy-85448bfccc-jdm4z\" (UID: \"3e5369da-d313-4e30-a3b4-cf31dfc43bbf\") " pod="openshift-ingress/kube-auth-proxy-85448bfccc-jdm4z" Nov 04 01:03:32.429100 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:32.429081 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e5369da-d313-4e30-a3b4-cf31dfc43bbf-tls-certs\") pod \"kube-auth-proxy-85448bfccc-jdm4z\" (UID: \"3e5369da-d313-4e30-a3b4-cf31dfc43bbf\") " pod="openshift-ingress/kube-auth-proxy-85448bfccc-jdm4z" Nov 04 01:03:32.435360 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:32.435338 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mvgt\" (UniqueName: \"kubernetes.io/projected/3e5369da-d313-4e30-a3b4-cf31dfc43bbf-kube-api-access-8mvgt\") pod \"kube-auth-proxy-85448bfccc-jdm4z\" (UID: \"3e5369da-d313-4e30-a3b4-cf31dfc43bbf\") " pod="openshift-ingress/kube-auth-proxy-85448bfccc-jdm4z" Nov 04 01:03:32.443067 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:32.443048 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85448bfccc-jdm4z" Nov 04 01:03:32.565325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:32.565294 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:32.715097 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:32.715067 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85448bfccc-jdm4z"] Nov 04 01:03:32.718166 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:03:32.718137 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5369da_d313_4e30_a3b4_cf31dfc43bbf.slice/crio-671426d7d29864a77cba8e71e299b7efd855f0815ec36d943d89d6f813b7d060 WatchSource:0}: Error finding container 671426d7d29864a77cba8e71e299b7efd855f0815ec36d943d89d6f813b7d060: Status 404 returned error can't find the container with id 671426d7d29864a77cba8e71e299b7efd855f0815ec36d943d89d6f813b7d060 Nov 04 01:03:33.043927 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:33.043890 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85448bfccc-jdm4z" event={"ID":"3e5369da-d313-4e30-a3b4-cf31dfc43bbf","Type":"ContainerStarted","Data":"671426d7d29864a77cba8e71e299b7efd855f0815ec36d943d89d6f813b7d060"} Nov 04 01:03:33.689920 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:33.689885 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:34.233579 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:34.233531 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:35.018529 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:35.018499 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b9nwg" Nov 04 01:03:36.061524 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:36.061485 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85448bfccc-jdm4z" event={"ID":"3e5369da-d313-4e30-a3b4-cf31dfc43bbf","Type":"ContainerStarted","Data":"b039ea875f36465c9c8ad025b54c1c9ad46ca8072d837cd16f91f51e34f8d7da"} Nov 04 01:03:36.079128 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:36.079072 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-85448bfccc-jdm4z" podStartSLOduration=0.912520452 podStartE2EDuration="4.079058815s" totalImagesPullingTime="3.166538363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:36.07791644 +0000 UTC m=+298.193026559" watchObservedRunningTime="2026-11-04 01:03:36.079058815 +0000 UTC m=+298.194168965" Nov 04 01:03:37.186831 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:37.186787 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:37.352951 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:37.352901 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:38.008169 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:38.008142 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 01:03:38.010580 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:38.010546 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 01:03:39.769633 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:39.769590 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:39.898664 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:39.898628 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:43.591104 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:43.591023 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:43.719253 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:43.719220 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:45.154644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:45.154602 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:45.287371 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:45.287339 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:45.562842 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:45.562779 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:45.697376 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:45.697315 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:49.772743 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:49.772678 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:49.909491 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:49.909448 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:56.215835 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:56.215800 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:56.798531 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:56.798496 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:56.942532 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:03:56.942499 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:59.865762 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:03:59.865721 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:04:28.369993 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:28.369952 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc"] Nov 04 01:04:28.373072 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:28.373055 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc" Nov 04 01:04:28.391396 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:28.391363 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc"] Nov 04 01:04:28.391396 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:28.391393 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-dg777\"" Nov 04 01:04:28.391672 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:28.391505 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 04 01:04:28.446257 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:28.446221 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 04 01:04:28.446443 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:28.446275 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 04 01:04:28.467744 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:28.467712 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:04:28.469832 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:28.469809 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zl8h\" (UniqueName: \"kubernetes.io/projected/d0dd4939-1fd6-42a7-8346-2c4228c632e9-kube-api-access-7zl8h\") pod \"llmisvc-controller-manager-869d5dcc5d-bnznc\" (UID: \"d0dd4939-1fd6-42a7-8346-2c4228c632e9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc" Nov 04 01:04:28.469916 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:28.469859 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0dd4939-1fd6-42a7-8346-2c4228c632e9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-bnznc\" (UID: \"d0dd4939-1fd6-42a7-8346-2c4228c632e9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc" Nov 04 01:04:28.571201 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:28.571163 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zl8h\" (UniqueName: \"kubernetes.io/projected/d0dd4939-1fd6-42a7-8346-2c4228c632e9-kube-api-access-7zl8h\") pod \"llmisvc-controller-manager-869d5dcc5d-bnznc\" (UID: \"d0dd4939-1fd6-42a7-8346-2c4228c632e9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc" Nov 04 01:04:28.571378 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:28.571229 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0dd4939-1fd6-42a7-8346-2c4228c632e9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-bnznc\" (UID: \"d0dd4939-1fd6-42a7-8346-2c4228c632e9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc" Nov 04 01:04:28.573824 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:28.573805 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0dd4939-1fd6-42a7-8346-2c4228c632e9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-bnznc\" (UID: \"d0dd4939-1fd6-42a7-8346-2c4228c632e9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc" Nov 04 01:04:28.582557 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:28.582533 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zl8h\" (UniqueName: \"kubernetes.io/projected/d0dd4939-1fd6-42a7-8346-2c4228c632e9-kube-api-access-7zl8h\") pod \"llmisvc-controller-manager-869d5dcc5d-bnznc\" (UID: \"d0dd4939-1fd6-42a7-8346-2c4228c632e9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc" Nov 04 01:04:28.684804 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:28.684703 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc" Nov 04 01:04:28.724092 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:28.724056 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:04:28.936591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:28.936453 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:04:28.957463 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:28.957431 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x"] Nov 04 01:04:28.961011 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:28.960991 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:28.974611 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:28.974586 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc"] Nov 04 01:04:28.977421 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:28.977396 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x"] Nov 04 01:04:28.978643 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:04:28.978592 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd0dd4939_1fd6_42a7_8346_2c4228c632e9.slice/crio-a64b9c01d01bfbfc26ec494cf6978743717415388cb2c4f1a60263f471d697e9 WatchSource:0}: Error finding container a64b9c01d01bfbfc26ec494cf6978743717415388cb2c4f1a60263f471d697e9: Status 404 returned error can't find the container with id a64b9c01d01bfbfc26ec494cf6978743717415388cb2c4f1a60263f471d697e9 Nov 04 01:04:28.980524 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:28.980502 4852 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:04:29.074853 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.074819 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f3466cf4-0baf-41e7-ab1a-263c895ab06d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9p2x\" (UID: \"f3466cf4-0baf-41e7-ab1a-263c895ab06d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:29.075053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.074877 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3466cf4-0baf-41e7-ab1a-263c895ab06d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9p2x\" (UID: \"f3466cf4-0baf-41e7-ab1a-263c895ab06d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:29.075053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.074955 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f3466cf4-0baf-41e7-ab1a-263c895ab06d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9p2x\" (UID: \"f3466cf4-0baf-41e7-ab1a-263c895ab06d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:29.075053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.075001 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p4lx\" (UniqueName: \"kubernetes.io/projected/f3466cf4-0baf-41e7-ab1a-263c895ab06d-kube-api-access-2p4lx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9p2x\" (UID: \"f3466cf4-0baf-41e7-ab1a-263c895ab06d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:29.075053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.075024 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f3466cf4-0baf-41e7-ab1a-263c895ab06d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9p2x\" (UID: \"f3466cf4-0baf-41e7-ab1a-263c895ab06d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:29.075053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.075046 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3466cf4-0baf-41e7-ab1a-263c895ab06d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9p2x\" (UID: \"f3466cf4-0baf-41e7-ab1a-263c895ab06d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:29.075251 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.075163 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f3466cf4-0baf-41e7-ab1a-263c895ab06d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9p2x\" (UID: \"f3466cf4-0baf-41e7-ab1a-263c895ab06d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:29.176277 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.176241 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f3466cf4-0baf-41e7-ab1a-263c895ab06d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9p2x\" (UID: \"f3466cf4-0baf-41e7-ab1a-263c895ab06d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:29.176445 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.176286 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f3466cf4-0baf-41e7-ab1a-263c895ab06d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9p2x\" (UID: \"f3466cf4-0baf-41e7-ab1a-263c895ab06d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:29.176445 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.176309 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3466cf4-0baf-41e7-ab1a-263c895ab06d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9p2x\" (UID: \"f3466cf4-0baf-41e7-ab1a-263c895ab06d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:29.176445 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.176419 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f3466cf4-0baf-41e7-ab1a-263c895ab06d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9p2x\" (UID: \"f3466cf4-0baf-41e7-ab1a-263c895ab06d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:29.176620 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.176461 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p4lx\" (UniqueName: \"kubernetes.io/projected/f3466cf4-0baf-41e7-ab1a-263c895ab06d-kube-api-access-2p4lx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9p2x\" (UID: \"f3466cf4-0baf-41e7-ab1a-263c895ab06d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:29.176620 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.176484 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f3466cf4-0baf-41e7-ab1a-263c895ab06d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9p2x\" (UID: \"f3466cf4-0baf-41e7-ab1a-263c895ab06d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:29.176620 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.176503 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3466cf4-0baf-41e7-ab1a-263c895ab06d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9p2x\" (UID: \"f3466cf4-0baf-41e7-ab1a-263c895ab06d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:29.177311 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.177281 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f3466cf4-0baf-41e7-ab1a-263c895ab06d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9p2x\" (UID: \"f3466cf4-0baf-41e7-ab1a-263c895ab06d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:29.178847 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.178824 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3466cf4-0baf-41e7-ab1a-263c895ab06d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9p2x\" (UID: \"f3466cf4-0baf-41e7-ab1a-263c895ab06d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:29.178940 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.178866 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f3466cf4-0baf-41e7-ab1a-263c895ab06d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9p2x\" (UID: \"f3466cf4-0baf-41e7-ab1a-263c895ab06d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:29.179264 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.179237 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f3466cf4-0baf-41e7-ab1a-263c895ab06d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9p2x\" (UID: \"f3466cf4-0baf-41e7-ab1a-263c895ab06d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:29.179337 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.179281 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f3466cf4-0baf-41e7-ab1a-263c895ab06d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9p2x\" (UID: \"f3466cf4-0baf-41e7-ab1a-263c895ab06d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:29.191636 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.191589 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3466cf4-0baf-41e7-ab1a-263c895ab06d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9p2x\" (UID: \"f3466cf4-0baf-41e7-ab1a-263c895ab06d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:29.191793 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.191776 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p4lx\" (UniqueName: \"kubernetes.io/projected/f3466cf4-0baf-41e7-ab1a-263c895ab06d-kube-api-access-2p4lx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c9p2x\" (UID: \"f3466cf4-0baf-41e7-ab1a-263c895ab06d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:29.261029 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.260997 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc" event={"ID":"d0dd4939-1fd6-42a7-8346-2c4228c632e9","Type":"ContainerStarted","Data":"a64b9c01d01bfbfc26ec494cf6978743717415388cb2c4f1a60263f471d697e9"} Nov 04 01:04:29.272224 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.272199 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:29.555771 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.555747 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x"] Nov 04 01:04:29.558239 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:04:29.558215 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3466cf4_0baf_41e7_ab1a_263c895ab06d.slice/crio-d6d756611a641da43f7ed9474bcbf5136cf22fe98f2c47e802e7e292ea429a15 WatchSource:0}: Error finding container d6d756611a641da43f7ed9474bcbf5136cf22fe98f2c47e802e7e292ea429a15: Status 404 returned error can't find the container with id d6d756611a641da43f7ed9474bcbf5136cf22fe98f2c47e802e7e292ea429a15 Nov 04 01:04:29.560414 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.560383 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:04:29.560473 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.560452 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:04:29.866412 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:04:29.866307 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:04:29.996486 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:29.996449 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:04:30.267548 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.267459 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" event={"ID":"f3466cf4-0baf-41e7-ab1a-263c895ab06d","Type":"ContainerStarted","Data":"6a4b0fb59a667fa80201a803a0809ae4a0d9cb5b220615bbc7db422e7ebb1a5f"} Nov 04 01:04:30.267548 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.267522 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" event={"ID":"f3466cf4-0baf-41e7-ab1a-263c895ab06d","Type":"ContainerStarted","Data":"d6d756611a641da43f7ed9474bcbf5136cf22fe98f2c47e802e7e292ea429a15"} Nov 04 01:04:30.267876 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.267684 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:30.270267 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.270242 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" Nov 04 01:04:30.271762 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.271727 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:04:30.290649 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.290592 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c9p2x" podStartSLOduration=2.29057784 podStartE2EDuration="2.29057784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:04:30.288862885 +0000 UTC m=+352.403973014" watchObservedRunningTime="2026-11-04 01:04:30.29057784 +0000 UTC m=+352.405687952" Nov 04 01:04:30.354849 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.354810 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w"] Nov 04 01:04:30.355102 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.355075 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" podUID="d95e9a2f-2434-44cc-91a0-4841f4470299" containerName="discovery" containerID="cri-o://9af1aaceebdcdf7573575bad99b46ae6ec3db0326b7cef94e540e002861047f2" gracePeriod=30 Nov 04 01:04:30.609961 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.609937 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:04:30.789166 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.789080 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d95e9a2f-2434-44cc-91a0-4841f4470299-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d95e9a2f-2434-44cc-91a0-4841f4470299-local-certs\") pod \"d95e9a2f-2434-44cc-91a0-4841f4470299\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " Nov 04 01:04:30.789166 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.789125 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95e9a2f-2434-44cc-91a0-4841f4470299-kube-api-access-j749r\" (UniqueName: \"kubernetes.io/projected/d95e9a2f-2434-44cc-91a0-4841f4470299-kube-api-access-j749r\") pod \"d95e9a2f-2434-44cc-91a0-4841f4470299\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " Nov 04 01:04:30.789403 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.789168 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-csr-dns-cert\") pod \"d95e9a2f-2434-44cc-91a0-4841f4470299\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " Nov 04 01:04:30.789403 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.789213 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-kubeconfig\") pod \"d95e9a2f-2434-44cc-91a0-4841f4470299\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " Nov 04 01:04:30.789403 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.789254 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-token\" (UniqueName: \"kubernetes.io/projected/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-token\") pod \"d95e9a2f-2434-44cc-91a0-4841f4470299\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " Nov 04 01:04:30.789403 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.789291 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-csr-ca-configmap\") pod \"d95e9a2f-2434-44cc-91a0-4841f4470299\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " Nov 04 01:04:30.789403 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.789340 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d95e9a2f-2434-44cc-91a0-4841f4470299-cacerts\" (UniqueName: \"kubernetes.io/secret/d95e9a2f-2434-44cc-91a0-4841f4470299-cacerts\") pod \"d95e9a2f-2434-44cc-91a0-4841f4470299\" (UID: \"d95e9a2f-2434-44cc-91a0-4841f4470299\") " Nov 04 01:04:30.789857 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.789785 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-csr-ca-configmap" pod "d95e9a2f-2434-44cc-91a0-4841f4470299" (UID: "d95e9a2f-2434-44cc-91a0-4841f4470299"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:04:30.792153 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.792110 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95e9a2f-2434-44cc-91a0-4841f4470299-local-certs" pod "d95e9a2f-2434-44cc-91a0-4841f4470299" (UID: "d95e9a2f-2434-44cc-91a0-4841f4470299"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:04:30.792332 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.792302 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-kubeconfig" pod "d95e9a2f-2434-44cc-91a0-4841f4470299" (UID: "d95e9a2f-2434-44cc-91a0-4841f4470299"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:04:30.792456 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.792434 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95e9a2f-2434-44cc-91a0-4841f4470299-cacerts" pod "d95e9a2f-2434-44cc-91a0-4841f4470299" (UID: "d95e9a2f-2434-44cc-91a0-4841f4470299"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:04:30.792939 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.792911 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95e9a2f-2434-44cc-91a0-4841f4470299-kube-api-access-j749r" pod "d95e9a2f-2434-44cc-91a0-4841f4470299" (UID: "d95e9a2f-2434-44cc-91a0-4841f4470299"). InnerVolumeSpecName "kube-api-access-j749r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:04:30.793165 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.793137 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-token" pod "d95e9a2f-2434-44cc-91a0-4841f4470299" (UID: "d95e9a2f-2434-44cc-91a0-4841f4470299"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:04:30.793288 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.793230 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-csr-dns-cert" pod "d95e9a2f-2434-44cc-91a0-4841f4470299" (UID: "d95e9a2f-2434-44cc-91a0-4841f4470299"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:04:30.890774 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.890720 4852 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-kubeconfig\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:04:30.890774 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.890767 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:04:30.890774 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.890782 4852 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-csr-ca-configmap\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:04:30.891037 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.890798 4852 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d95e9a2f-2434-44cc-91a0-4841f4470299-cacerts\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:04:30.891037 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.890811 4852 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d95e9a2f-2434-44cc-91a0-4841f4470299-local-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:04:30.891037 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.890823 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j749r\" (UniqueName: \"kubernetes.io/projected/d95e9a2f-2434-44cc-91a0-4841f4470299-kube-api-access-j749r\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:04:30.891037 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:30.890835 4852 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d95e9a2f-2434-44cc-91a0-4841f4470299-istio-csr-dns-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:04:31.274859 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:31.274823 4852 generic.go:356] "Generic (PLEG): container finished" podID="d95e9a2f-2434-44cc-91a0-4841f4470299" containerID="9af1aaceebdcdf7573575bad99b46ae6ec3db0326b7cef94e540e002861047f2" exitCode=0 Nov 04 01:04:31.275042 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:31.274878 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" Nov 04 01:04:31.275042 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:31.274916 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" event={"ID":"d95e9a2f-2434-44cc-91a0-4841f4470299","Type":"ContainerDied","Data":"9af1aaceebdcdf7573575bad99b46ae6ec3db0326b7cef94e540e002861047f2"} Nov 04 01:04:31.275042 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:31.274979 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w" event={"ID":"d95e9a2f-2434-44cc-91a0-4841f4470299","Type":"ContainerDied","Data":"514f4faf9ed025bf8bacad340ce5a9c5c24c34e6d150c1dd41f66eff39ce1999"} Nov 04 01:04:31.275042 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:31.275026 4852 scope.go:122] "RemoveContainer" containerID="9af1aaceebdcdf7573575bad99b46ae6ec3db0326b7cef94e540e002861047f2" Nov 04 01:04:31.285809 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:31.285789 4852 scope.go:122] "RemoveContainer" containerID="9af1aaceebdcdf7573575bad99b46ae6ec3db0326b7cef94e540e002861047f2" Nov 04 01:04:31.286146 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:04:31.286121 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af1aaceebdcdf7573575bad99b46ae6ec3db0326b7cef94e540e002861047f2\": container with ID starting with 9af1aaceebdcdf7573575bad99b46ae6ec3db0326b7cef94e540e002861047f2 not found: ID does not exist" containerID="9af1aaceebdcdf7573575bad99b46ae6ec3db0326b7cef94e540e002861047f2" Nov 04 01:04:31.286191 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:31.286158 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af1aaceebdcdf7573575bad99b46ae6ec3db0326b7cef94e540e002861047f2"} err="failed to get container status \"9af1aaceebdcdf7573575bad99b46ae6ec3db0326b7cef94e540e002861047f2\": rpc error: code = NotFound desc = could not find container \"9af1aaceebdcdf7573575bad99b46ae6ec3db0326b7cef94e540e002861047f2\": container with ID starting with 9af1aaceebdcdf7573575bad99b46ae6ec3db0326b7cef94e540e002861047f2 not found: ID does not exist" Nov 04 01:04:31.298295 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:31.298270 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w"] Nov 04 01:04:31.304724 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:31.304669 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l6g9w"] Nov 04 01:04:31.997422 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:31.997386 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d95e9a2f-2434-44cc-91a0-4841f4470299" path="/var/lib/kubelet/pods/d95e9a2f-2434-44cc-91a0-4841f4470299/volumes" Nov 04 01:04:33.285642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:33.285596 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc" event={"ID":"d0dd4939-1fd6-42a7-8346-2c4228c632e9","Type":"ContainerStarted","Data":"39d0612b1d16161e864aa18fb33d8ef6d0f49dc07e3278d6269d1608c1f066cc"} Nov 04 01:04:33.286098 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:33.285663 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc" Nov 04 01:04:33.303993 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:33.303938 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc" podStartSLOduration=1.228786151 podStartE2EDuration="5.303924027s" totalImagesPullingTime="4.075137876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:04:33.303127388 +0000 UTC m=+355.418237505" watchObservedRunningTime="2026-11-04 01:04:33.303924027 +0000 UTC m=+355.419034144" Nov 04 01:04:37.661671 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:04:37.661631 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:04:59.865206 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:04:59.865165 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:05:04.293534 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:05:04.293504 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc" Nov 04 01:05:12.656937 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:05:12.656848 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:05:29.865664 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:05:29.865627 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:05:30.093954 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:05:30.093912 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:05:45.830787 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:05:45.830751 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:05:46.212929 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:05:46.212804 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:05:59.864103 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:05:59.864069 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:06:06.893429 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:06:06.893393 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:06:07.032928 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:06:07.032891 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:06:29.866133 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:06:29.866088 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:06:30.011786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:06:30.011739 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:06:59.865976 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:06:59.865895 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:07:29.865396 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:07:29.865354 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:07:30.061988 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:07:30.061956 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:07:59.865591 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:07:59.865537 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:08:29.864857 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:08:29.864778 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:08:30.045945 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:08:30.045916 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:08:38.035455 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:08:38.035429 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 01:08:38.039913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:08:38.039893 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 01:08:51.920580 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:08:51.920529 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:08:59.865987 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:08:59.865944 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:09:29.865777 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:09:29.865743 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:09:30.156457 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:09:30.156383 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:09:59.864456 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:09:59.864374 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:10:29.864900 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:10:29.864865 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:10:30.189576 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:10:30.189484 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:10:59.864481 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:10:59.864425 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:11:29.866664 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:11:29.866583 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:11:30.239702 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:11:30.239625 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:11:59.865209 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:11:59.865173 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:12:29.864554 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:12:29.864516 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:12:30.282476 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:12:30.282443 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:12:59.865017 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:12:59.864982 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:13:29.865356 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:13:29.865316 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:13:30.324406 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:13:30.324373 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:13:38.060960 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:13:38.060935 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 01:13:38.065695 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:13:38.065674 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 01:13:44.037187 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:13:44.037151 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:13:59.865704 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:13:59.865664 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:14:08.727931 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:08.727889 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:14:08.931935 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:08.931906 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:14:09.082994 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:09.082962 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:14:09.083312 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:09.083298 4852 cpu_manager.go:414] "containerMap: removing container" podUID="d95e9a2f-2434-44cc-91a0-4841f4470299" containerName="discovery" Nov 04 01:14:09.083371 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:09.083315 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95e9a2f-2434-44cc-91a0-4841f4470299" containerName="discovery" Nov 04 01:14:09.083406 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:09.083376 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95e9a2f-2434-44cc-91a0-4841f4470299" containerName="discovery" Nov 04 01:14:09.086398 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:09.086376 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-8r2fx" Nov 04 01:14:09.091004 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:09.090985 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8gbdw\"" Nov 04 01:14:09.094923 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:09.094877 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:14:09.107697 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:09.107676 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 04 01:14:09.108017 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:09.108000 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:14:09.140711 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:09.140678 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtj7x\" (UniqueName: \"kubernetes.io/projected/8324a6d0-9083-4288-afb5-63729b6430d6-kube-api-access-gtj7x\") pod \"limitador-limitador-7c9b87d574-8r2fx\" (UID: \"8324a6d0-9083-4288-afb5-63729b6430d6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8r2fx" Nov 04 01:14:09.140855 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:09.140717 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8324a6d0-9083-4288-afb5-63729b6430d6-config-file\") pod \"limitador-limitador-7c9b87d574-8r2fx\" (UID: \"8324a6d0-9083-4288-afb5-63729b6430d6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8r2fx" Nov 04 01:14:09.241713 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:09.241683 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtj7x\" (UniqueName: \"kubernetes.io/projected/8324a6d0-9083-4288-afb5-63729b6430d6-kube-api-access-gtj7x\") pod \"limitador-limitador-7c9b87d574-8r2fx\" (UID: \"8324a6d0-9083-4288-afb5-63729b6430d6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8r2fx" Nov 04 01:14:09.241867 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:09.241718 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8324a6d0-9083-4288-afb5-63729b6430d6-config-file\") pod \"limitador-limitador-7c9b87d574-8r2fx\" (UID: \"8324a6d0-9083-4288-afb5-63729b6430d6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8r2fx" Nov 04 01:14:09.242318 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:09.242300 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8324a6d0-9083-4288-afb5-63729b6430d6-config-file\") pod \"limitador-limitador-7c9b87d574-8r2fx\" (UID: \"8324a6d0-9083-4288-afb5-63729b6430d6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8r2fx" Nov 04 01:14:09.249632 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:09.249606 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtj7x\" (UniqueName: \"kubernetes.io/projected/8324a6d0-9083-4288-afb5-63729b6430d6-kube-api-access-gtj7x\") pod \"limitador-limitador-7c9b87d574-8r2fx\" (UID: \"8324a6d0-9083-4288-afb5-63729b6430d6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8r2fx" Nov 04 01:14:09.399262 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:09.399176 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-8r2fx" Nov 04 01:14:09.563297 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:09.563267 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:14:09.671018 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:09.670943 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:14:09.674164 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:14:09.674138 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8324a6d0_9083_4288_afb5_63729b6430d6.slice/crio-bfba47acc5b18ac3688712d6fc90cb69f07aaf27ea4c0eee45745490b4018348 WatchSource:0}: Error finding container bfba47acc5b18ac3688712d6fc90cb69f07aaf27ea4c0eee45745490b4018348: Status 404 returned error can't find the container with id bfba47acc5b18ac3688712d6fc90cb69f07aaf27ea4c0eee45745490b4018348 Nov 04 01:14:09.676608 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:09.676590 4852 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:14:09.810819 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:09.810788 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:14:10.477839 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:10.477804 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-8r2fx" event={"ID":"8324a6d0-9083-4288-afb5-63729b6430d6","Type":"ContainerStarted","Data":"bfba47acc5b18ac3688712d6fc90cb69f07aaf27ea4c0eee45745490b4018348"} Nov 04 01:14:13.491606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:13.491488 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-8r2fx" event={"ID":"8324a6d0-9083-4288-afb5-63729b6430d6","Type":"ContainerStarted","Data":"5ce96b7c952c7424f1d62365d2b5a7e73e875344968aaf916f78a6a99741be4f"} Nov 04 01:14:13.491998 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:13.491717 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-8r2fx" Nov 04 01:14:13.508346 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:13.508292 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-8r2fx" podStartSLOduration=1.138286423 podStartE2EDuration="4.50827858s" totalImagesPullingTime="3.369992157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:13.507058224 +0000 UTC m=+935.622168341" watchObservedRunningTime="2026-11-04 01:14:13.50827858 +0000 UTC m=+935.623388700" Nov 04 01:14:24.495978 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:24.495947 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-8r2fx" Nov 04 01:14:26.548717 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:26.548681 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:14:26.701209 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:26.701176 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:14:27.307805 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:27.307771 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:14:29.864973 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:14:29.864938 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:14:30.367087 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:30.367055 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:14:40.943194 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:40.943153 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:14:47.046898 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:47.046865 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:14:47.189633 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:47.189602 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:14:47.321379 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:47.321298 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:14:51.131376 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:51.131345 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:14:51.298185 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:51.298150 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:14:52.687074 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:52.687043 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:14:52.822677 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:14:52.822637 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:14:59.864878 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:14:59.864831 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:15:21.406798 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:21.406758 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:15:21.661926 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:21.661841 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:15:22.739948 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:22.739915 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:15:22.932730 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:22.932699 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:15:23.073853 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:23.073819 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:15:23.559991 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:23.559951 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:15:29.864849 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:15:29.864814 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:15:30.425061 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:30.425030 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:15:32.638273 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:32.638239 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:15:32.848935 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:32.848898 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:15:33.087686 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.087655 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6"] Nov 04 01:15:33.090949 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.090933 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" Nov 04 01:15:33.095666 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.095642 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nb79d\"" Nov 04 01:15:33.095792 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.095672 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 01:15:33.095792 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.095723 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 04 01:15:33.100222 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.100199 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6"] Nov 04 01:15:33.102948 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.102931 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:15:33.102948 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.102931 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:15:33.196318 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.196289 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-maas-api-tls\") pod \"maas-api-847bf86b74-6mfj6\" (UID: \"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" Nov 04 01:15:33.196500 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.196328 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-metrics-certs\") pod \"maas-api-847bf86b74-6mfj6\" (UID: \"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" Nov 04 01:15:33.196500 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.196356 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-openshift-service-ca\") pod \"maas-api-847bf86b74-6mfj6\" (UID: \"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" Nov 04 01:15:33.196500 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.196438 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjs6j\" (UniqueName: \"kubernetes.io/projected/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-kube-api-access-vjs6j\") pod \"maas-api-847bf86b74-6mfj6\" (UID: \"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" Nov 04 01:15:33.297637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.297601 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-maas-api-tls\") pod \"maas-api-847bf86b74-6mfj6\" (UID: \"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" Nov 04 01:15:33.297805 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.297655 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-metrics-certs\") pod \"maas-api-847bf86b74-6mfj6\" (UID: \"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" Nov 04 01:15:33.297927 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.297906 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-openshift-service-ca\") pod \"maas-api-847bf86b74-6mfj6\" (UID: \"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" Nov 04 01:15:33.297981 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.297943 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjs6j\" (UniqueName: \"kubernetes.io/projected/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-kube-api-access-vjs6j\") pod \"maas-api-847bf86b74-6mfj6\" (UID: \"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" Nov 04 01:15:33.298513 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.298486 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-openshift-service-ca\") pod \"maas-api-847bf86b74-6mfj6\" (UID: \"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" Nov 04 01:15:33.300131 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.300108 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-maas-api-tls\") pod \"maas-api-847bf86b74-6mfj6\" (UID: \"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" Nov 04 01:15:33.300377 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.300358 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-metrics-certs\") pod \"maas-api-847bf86b74-6mfj6\" (UID: \"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" Nov 04 01:15:33.309391 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.309367 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjs6j\" (UniqueName: \"kubernetes.io/projected/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-kube-api-access-vjs6j\") pod \"maas-api-847bf86b74-6mfj6\" (UID: \"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" Nov 04 01:15:33.404310 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.404238 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" Nov 04 01:15:33.676199 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.676135 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6"] Nov 04 01:15:33.678758 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:15:33.678733 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a6c8f1_0a9c_4c04_805b_a8b6ad539fef.slice/crio-f4bac244a51faa793cc4fcba4088219e6a0cb95a26be99dfdee6684b13c5ef46 WatchSource:0}: Error finding container f4bac244a51faa793cc4fcba4088219e6a0cb95a26be99dfdee6684b13c5ef46: Status 404 returned error can't find the container with id f4bac244a51faa793cc4fcba4088219e6a0cb95a26be99dfdee6684b13c5ef46 Nov 04 01:15:33.814610 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:33.814578 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" event={"ID":"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef","Type":"ContainerStarted","Data":"f4bac244a51faa793cc4fcba4088219e6a0cb95a26be99dfdee6684b13c5ef46"} Nov 04 01:15:36.159361 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:36.159339 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-k8zn6"] Nov 04 01:15:36.163886 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:36.163870 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-k8zn6" Nov 04 01:15:36.172394 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:36.172341 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-k8zn6"] Nov 04 01:15:36.326064 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:36.326027 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c322dc68-39b6-40fc-8a6a-ebe9867f1d9c-cert\") pod \"llmisvc-controller-manager-6466c554d8-k8zn6\" (UID: \"c322dc68-39b6-40fc-8a6a-ebe9867f1d9c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-k8zn6" Nov 04 01:15:36.326253 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:36.326151 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxtpm\" (UniqueName: \"kubernetes.io/projected/c322dc68-39b6-40fc-8a6a-ebe9867f1d9c-kube-api-access-pxtpm\") pod \"llmisvc-controller-manager-6466c554d8-k8zn6\" (UID: \"c322dc68-39b6-40fc-8a6a-ebe9867f1d9c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-k8zn6" Nov 04 01:15:36.431476 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:36.427145 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxtpm\" (UniqueName: \"kubernetes.io/projected/c322dc68-39b6-40fc-8a6a-ebe9867f1d9c-kube-api-access-pxtpm\") pod \"llmisvc-controller-manager-6466c554d8-k8zn6\" (UID: \"c322dc68-39b6-40fc-8a6a-ebe9867f1d9c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-k8zn6" Nov 04 01:15:36.431476 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:36.427240 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c322dc68-39b6-40fc-8a6a-ebe9867f1d9c-cert\") pod \"llmisvc-controller-manager-6466c554d8-k8zn6\" (UID: \"c322dc68-39b6-40fc-8a6a-ebe9867f1d9c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-k8zn6" Nov 04 01:15:36.433628 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:36.433594 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c322dc68-39b6-40fc-8a6a-ebe9867f1d9c-cert\") pod \"llmisvc-controller-manager-6466c554d8-k8zn6\" (UID: \"c322dc68-39b6-40fc-8a6a-ebe9867f1d9c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-k8zn6" Nov 04 01:15:36.436527 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:36.436503 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxtpm\" (UniqueName: \"kubernetes.io/projected/c322dc68-39b6-40fc-8a6a-ebe9867f1d9c-kube-api-access-pxtpm\") pod \"llmisvc-controller-manager-6466c554d8-k8zn6\" (UID: \"c322dc68-39b6-40fc-8a6a-ebe9867f1d9c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-k8zn6" Nov 04 01:15:36.496881 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:36.496846 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-k8zn6" Nov 04 01:15:36.770454 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:36.770414 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-k8zn6"] Nov 04 01:15:36.774374 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:15:36.774349 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc322dc68_39b6_40fc_8a6a_ebe9867f1d9c.slice/crio-c99d4ac4626b60bbe32d9352c2bee9cb64bacaa984deab452994039934f608d5 WatchSource:0}: Error finding container c99d4ac4626b60bbe32d9352c2bee9cb64bacaa984deab452994039934f608d5: Status 404 returned error can't find the container with id c99d4ac4626b60bbe32d9352c2bee9cb64bacaa984deab452994039934f608d5 Nov 04 01:15:36.827516 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:36.827483 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-k8zn6" event={"ID":"c322dc68-39b6-40fc-8a6a-ebe9867f1d9c","Type":"ContainerStarted","Data":"c99d4ac4626b60bbe32d9352c2bee9cb64bacaa984deab452994039934f608d5"} Nov 04 01:15:36.828910 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:36.828882 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" event={"ID":"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef","Type":"ContainerStarted","Data":"c33c6360edd18b094fb008f28eb8e3f514ecbe683bd652482aaaf41a4e7d857e"} Nov 04 01:15:36.829035 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:36.828988 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" Nov 04 01:15:36.847250 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:36.847209 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" podStartSLOduration=1.390007273 podStartE2EDuration="3.84719741s" totalImagesPullingTime="2.457190137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:36.846233872 +0000 UTC m=+1018.961343993" watchObservedRunningTime="2026-11-04 01:15:36.84719741 +0000 UTC m=+1018.962307547" Nov 04 01:15:37.834372 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:37.834331 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-k8zn6" event={"ID":"c322dc68-39b6-40fc-8a6a-ebe9867f1d9c","Type":"ContainerStarted","Data":"6d561e9e37b0a9f4952b3ce808fef0e5385df6f00c61e610d8e6a0b9e443bf6a"} Nov 04 01:15:37.851744 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:37.851701 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-k8zn6" podStartSLOduration=1.371004748 podStartE2EDuration="1.851686864s" totalImagesPullingTime="480.682116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:37.849848987 +0000 UTC m=+1019.964959097" watchObservedRunningTime="2026-11-04 01:15:37.851686864 +0000 UTC m=+1019.966797027" Nov 04 01:15:38.839398 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:38.839366 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-k8zn6" Nov 04 01:15:42.839624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:15:42.839534 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" Nov 04 01:15:59.864797 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:15:59.864752 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:16:09.850746 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:09.850716 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-k8zn6" Nov 04 01:16:09.894534 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:09.894500 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc"] Nov 04 01:16:09.894758 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:09.894733 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc" podUID="d0dd4939-1fd6-42a7-8346-2c4228c632e9" containerName="manager" containerID="cri-o://39d0612b1d16161e864aa18fb33d8ef6d0f49dc07e3278d6269d1608c1f066cc" gracePeriod=30 Nov 04 01:16:10.145604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.145582 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc" Nov 04 01:16:10.181402 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.181374 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0dd4939-1fd6-42a7-8346-2c4228c632e9-cert\" (UniqueName: \"kubernetes.io/secret/d0dd4939-1fd6-42a7-8346-2c4228c632e9-cert\") pod \"d0dd4939-1fd6-42a7-8346-2c4228c632e9\" (UID: \"d0dd4939-1fd6-42a7-8346-2c4228c632e9\") " Nov 04 01:16:10.181605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.181416 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0dd4939-1fd6-42a7-8346-2c4228c632e9-kube-api-access-7zl8h\" (UniqueName: \"kubernetes.io/projected/d0dd4939-1fd6-42a7-8346-2c4228c632e9-kube-api-access-7zl8h\") pod \"d0dd4939-1fd6-42a7-8346-2c4228c632e9\" (UID: \"d0dd4939-1fd6-42a7-8346-2c4228c632e9\") " Nov 04 01:16:10.183783 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.183754 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0dd4939-1fd6-42a7-8346-2c4228c632e9-cert" pod "d0dd4939-1fd6-42a7-8346-2c4228c632e9" (UID: "d0dd4939-1fd6-42a7-8346-2c4228c632e9"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:16:10.183883 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.183817 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0dd4939-1fd6-42a7-8346-2c4228c632e9-kube-api-access-7zl8h" pod "d0dd4939-1fd6-42a7-8346-2c4228c632e9" (UID: "d0dd4939-1fd6-42a7-8346-2c4228c632e9"). InnerVolumeSpecName "kube-api-access-7zl8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:16:10.282654 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.282625 4852 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0dd4939-1fd6-42a7-8346-2c4228c632e9-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:10.282654 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.282651 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zl8h\" (UniqueName: \"kubernetes.io/projected/d0dd4939-1fd6-42a7-8346-2c4228c632e9-kube-api-access-7zl8h\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:10.591992 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.591950 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk"] Nov 04 01:16:10.592254 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.592240 4852 cpu_manager.go:414] "containerMap: removing container" podUID="d0dd4939-1fd6-42a7-8346-2c4228c632e9" containerName="manager" Nov 04 01:16:10.592305 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.592255 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0dd4939-1fd6-42a7-8346-2c4228c632e9" containerName="manager" Nov 04 01:16:10.592339 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.592319 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0dd4939-1fd6-42a7-8346-2c4228c632e9" containerName="manager" Nov 04 01:16:10.595529 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.595508 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.609301 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.609272 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk"] Nov 04 01:16:10.683798 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.683772 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-f28wr\"" Nov 04 01:16:10.684964 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.684943 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266b7c8e-ffa2-410b-a592-a96f3e034103-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.685064 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.684989 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqt5g\" (UniqueName: \"kubernetes.io/projected/266b7c8e-ffa2-410b-a592-a96f3e034103-kube-api-access-gqt5g\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.685064 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.685014 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266b7c8e-ffa2-410b-a592-a96f3e034103-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.685172 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.685086 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266b7c8e-ffa2-410b-a592-a96f3e034103-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.685172 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.685142 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266b7c8e-ffa2-410b-a592-a96f3e034103-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.685272 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.685193 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266b7c8e-ffa2-410b-a592-a96f3e034103-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.685272 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.685257 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266b7c8e-ffa2-410b-a592-a96f3e034103-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.685339 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.685293 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/266b7c8e-ffa2-410b-a592-a96f3e034103-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.685339 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.685313 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266b7c8e-ffa2-410b-a592-a96f3e034103-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.785948 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.785923 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266b7c8e-ffa2-410b-a592-a96f3e034103-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.786053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.785955 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266b7c8e-ffa2-410b-a592-a96f3e034103-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.786053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.785975 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266b7c8e-ffa2-410b-a592-a96f3e034103-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.786053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.785996 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/266b7c8e-ffa2-410b-a592-a96f3e034103-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.786053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.786016 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266b7c8e-ffa2-410b-a592-a96f3e034103-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.786053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.786045 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266b7c8e-ffa2-410b-a592-a96f3e034103-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.786246 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.786079 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqt5g\" (UniqueName: \"kubernetes.io/projected/266b7c8e-ffa2-410b-a592-a96f3e034103-kube-api-access-gqt5g\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.786246 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.786108 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266b7c8e-ffa2-410b-a592-a96f3e034103-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.786246 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.786138 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266b7c8e-ffa2-410b-a592-a96f3e034103-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.786464 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.786415 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266b7c8e-ffa2-410b-a592-a96f3e034103-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.786464 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.786425 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266b7c8e-ffa2-410b-a592-a96f3e034103-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.786670 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.786543 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266b7c8e-ffa2-410b-a592-a96f3e034103-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.786747 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.786730 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266b7c8e-ffa2-410b-a592-a96f3e034103-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.786913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.786873 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266b7c8e-ffa2-410b-a592-a96f3e034103-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.787675 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.787656 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:16:10.788721 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.788700 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266b7c8e-ffa2-410b-a592-a96f3e034103-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.788866 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.788850 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266b7c8e-ffa2-410b-a592-a96f3e034103-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.794264 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.794247 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/266b7c8e-ffa2-410b-a592-a96f3e034103-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.794716 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.794701 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqt5g\" (UniqueName: \"kubernetes.io/projected/266b7c8e-ffa2-410b-a592-a96f3e034103-kube-api-access-gqt5g\") pod \"maas-default-gateway-openshift-default-75d7c785f-2ttwk\" (UID: \"266b7c8e-ffa2-410b-a592-a96f3e034103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.909704 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.909658 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:10.939330 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.939309 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:16:10.979643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.979613 4852 generic.go:356] "Generic (PLEG): container finished" podID="d0dd4939-1fd6-42a7-8346-2c4228c632e9" containerID="39d0612b1d16161e864aa18fb33d8ef6d0f49dc07e3278d6269d1608c1f066cc" exitCode=0 Nov 04 01:16:10.979771 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.979705 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc" event={"ID":"d0dd4939-1fd6-42a7-8346-2c4228c632e9","Type":"ContainerDied","Data":"39d0612b1d16161e864aa18fb33d8ef6d0f49dc07e3278d6269d1608c1f066cc"} Nov 04 01:16:10.979771 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.979758 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc" event={"ID":"d0dd4939-1fd6-42a7-8346-2c4228c632e9","Type":"ContainerDied","Data":"a64b9c01d01bfbfc26ec494cf6978743717415388cb2c4f1a60263f471d697e9"} Nov 04 01:16:10.979886 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.979791 4852 scope.go:122] "RemoveContainer" containerID="39d0612b1d16161e864aa18fb33d8ef6d0f49dc07e3278d6269d1608c1f066cc" Nov 04 01:16:10.979972 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.979955 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc" Nov 04 01:16:10.991864 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.991847 4852 scope.go:122] "RemoveContainer" containerID="39d0612b1d16161e864aa18fb33d8ef6d0f49dc07e3278d6269d1608c1f066cc" Nov 04 01:16:10.992104 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:16:10.992087 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39d0612b1d16161e864aa18fb33d8ef6d0f49dc07e3278d6269d1608c1f066cc\": container with ID starting with 39d0612b1d16161e864aa18fb33d8ef6d0f49dc07e3278d6269d1608c1f066cc not found: ID does not exist" containerID="39d0612b1d16161e864aa18fb33d8ef6d0f49dc07e3278d6269d1608c1f066cc" Nov 04 01:16:10.992165 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.992112 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39d0612b1d16161e864aa18fb33d8ef6d0f49dc07e3278d6269d1608c1f066cc"} err="failed to get container status \"39d0612b1d16161e864aa18fb33d8ef6d0f49dc07e3278d6269d1608c1f066cc\": rpc error: code = NotFound desc = could not find container \"39d0612b1d16161e864aa18fb33d8ef6d0f49dc07e3278d6269d1608c1f066cc\": container with ID starting with 39d0612b1d16161e864aa18fb33d8ef6d0f49dc07e3278d6269d1608c1f066cc not found: ID does not exist" Nov 04 01:16:10.999762 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:10.999743 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc"] Nov 04 01:16:11.006414 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:11.006393 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bnznc"] Nov 04 01:16:11.186946 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:11.186927 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk"] Nov 04 01:16:11.189242 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:16:11.189221 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266b7c8e_ffa2_410b_a592_a96f3e034103.slice/crio-7aca1900cc0498cd378ab5390ba1cd3498b13924c3bb182f8652444437e0760d WatchSource:0}: Error finding container 7aca1900cc0498cd378ab5390ba1cd3498b13924c3bb182f8652444437e0760d: Status 404 returned error can't find the container with id 7aca1900cc0498cd378ab5390ba1cd3498b13924c3bb182f8652444437e0760d Nov 04 01:16:11.995019 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:11.994986 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0dd4939-1fd6-42a7-8346-2c4228c632e9" path="/var/lib/kubelet/pods/d0dd4939-1fd6-42a7-8346-2c4228c632e9/volumes" Nov 04 01:16:11.995491 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:11.995235 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" event={"ID":"266b7c8e-ffa2-410b-a592-a96f3e034103","Type":"ContainerStarted","Data":"7aca1900cc0498cd378ab5390ba1cd3498b13924c3bb182f8652444437e0760d"} Nov 04 01:16:14.525753 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:14.525722 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:16:14.526009 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:14.525788 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:16:14.526009 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:14.525818 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:16:15.017352 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:15.017319 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" event={"ID":"266b7c8e-ffa2-410b-a592-a96f3e034103","Type":"ContainerStarted","Data":"c47584bf524a7022c3535c1474d58ab41a0e5d48f8042951b1edc2771797aa7e"} Nov 04 01:16:15.040086 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:15.040044 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" podStartSLOduration=1.706221757 podStartE2EDuration="5.040030758s" totalImagesPullingTime="3.333809001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:15.038271829 +0000 UTC m=+1057.153381951" watchObservedRunningTime="2026-11-04 01:16:15.040030758 +0000 UTC m=+1057.155140876" Nov 04 01:16:15.910626 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:15.910554 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:15.912079 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:15.912054 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2ttwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Nov 04 01:16:15.912197 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:15.912104 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" podUID="266b7c8e-ffa2-410b-a592-a96f3e034103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Nov 04 01:16:16.911118 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:16.911087 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2ttwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Nov 04 01:16:16.911482 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:16.911155 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" podUID="266b7c8e-ffa2-410b-a592-a96f3e034103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Nov 04 01:16:17.910480 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:17.910447 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2ttwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Nov 04 01:16:17.910661 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:17.910508 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" podUID="266b7c8e-ffa2-410b-a592-a96f3e034103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Nov 04 01:16:18.911000 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:18.910961 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2ttwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Nov 04 01:16:18.911403 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:18.911016 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" podUID="266b7c8e-ffa2-410b-a592-a96f3e034103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Nov 04 01:16:19.910720 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:19.910688 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2ttwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Nov 04 01:16:19.910898 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:19.910752 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" podUID="266b7c8e-ffa2-410b-a592-a96f3e034103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Nov 04 01:16:20.913977 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:20.913946 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:20.914392 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:20.914231 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:20.914906 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:20.914891 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" Nov 04 01:16:25.614659 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.614628 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84"] Nov 04 01:16:25.618022 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.617999 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.634195 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.633646 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84"] Nov 04 01:16:25.728303 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.728277 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13769b4-f82d-460e-aa0f-0e882b40346c-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.728444 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.728310 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13769b4-f82d-460e-aa0f-0e882b40346c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.728444 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.728418 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13769b4-f82d-460e-aa0f-0e882b40346c-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.728516 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.728456 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkxcl\" (UniqueName: \"kubernetes.io/projected/c13769b4-f82d-460e-aa0f-0e882b40346c-kube-api-access-hkxcl\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.728549 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.728520 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13769b4-f82d-460e-aa0f-0e882b40346c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.728603 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.728556 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13769b4-f82d-460e-aa0f-0e882b40346c-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.728668 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.728655 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13769b4-f82d-460e-aa0f-0e882b40346c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.728712 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.728674 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13769b4-f82d-460e-aa0f-0e882b40346c-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.728712 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.728695 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13769b4-f82d-460e-aa0f-0e882b40346c-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.829337 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.829312 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13769b4-f82d-460e-aa0f-0e882b40346c-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.829440 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.829341 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13769b4-f82d-460e-aa0f-0e882b40346c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.829440 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.829378 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13769b4-f82d-460e-aa0f-0e882b40346c-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.829440 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.829396 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkxcl\" (UniqueName: \"kubernetes.io/projected/c13769b4-f82d-460e-aa0f-0e882b40346c-kube-api-access-hkxcl\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.829440 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.829422 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13769b4-f82d-460e-aa0f-0e882b40346c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.829662 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.829449 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13769b4-f82d-460e-aa0f-0e882b40346c-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.829662 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.829480 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13769b4-f82d-460e-aa0f-0e882b40346c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.829662 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.829499 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13769b4-f82d-460e-aa0f-0e882b40346c-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.829662 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.829527 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13769b4-f82d-460e-aa0f-0e882b40346c-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.829857 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.829787 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13769b4-f82d-460e-aa0f-0e882b40346c-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.829930 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.829911 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13769b4-f82d-460e-aa0f-0e882b40346c-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.829985 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.829950 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13769b4-f82d-460e-aa0f-0e882b40346c-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.830033 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.829979 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13769b4-f82d-460e-aa0f-0e882b40346c-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.830112 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.830091 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13769b4-f82d-460e-aa0f-0e882b40346c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.831804 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.831784 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13769b4-f82d-460e-aa0f-0e882b40346c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.832230 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.832213 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13769b4-f82d-460e-aa0f-0e882b40346c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.837714 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.837691 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13769b4-f82d-460e-aa0f-0e882b40346c-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.837806 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.837716 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkxcl\" (UniqueName: \"kubernetes.io/projected/c13769b4-f82d-460e-aa0f-0e882b40346c-kube-api-access-hkxcl\") pod \"maas-default-gateway-openshift-default-75d7c785f-tqg84\" (UID: \"c13769b4-f82d-460e-aa0f-0e882b40346c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:25.930618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:25.930544 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:26.207701 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:26.207677 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84"] Nov 04 01:16:26.210331 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:16:26.210297 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13769b4_f82d_460e_aa0f_0e882b40346c.slice/crio-60350ebaf23d3bf3be53d885ed8ad79208b7c8223912f0f00b0fa0cf55acd36b WatchSource:0}: Error finding container 60350ebaf23d3bf3be53d885ed8ad79208b7c8223912f0f00b0fa0cf55acd36b: Status 404 returned error can't find the container with id 60350ebaf23d3bf3be53d885ed8ad79208b7c8223912f0f00b0fa0cf55acd36b Nov 04 01:16:26.213257 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:26.213225 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:16:26.213359 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:26.213302 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:16:26.213359 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:26.213347 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:16:27.083652 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:27.083617 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" event={"ID":"c13769b4-f82d-460e-aa0f-0e882b40346c","Type":"ContainerStarted","Data":"40bff525f2051fe0ae2196f81a207dc7572bff4ca3afe4286bf46fae284a3a99"} Nov 04 01:16:27.084043 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:27.083673 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" event={"ID":"c13769b4-f82d-460e-aa0f-0e882b40346c","Type":"ContainerStarted","Data":"60350ebaf23d3bf3be53d885ed8ad79208b7c8223912f0f00b0fa0cf55acd36b"} Nov 04 01:16:27.105529 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:27.105469 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" podStartSLOduration=2.105451949 podStartE2EDuration="2.105451949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:27.103800958 +0000 UTC m=+1069.218911075" watchObservedRunningTime="2026-11-04 01:16:27.105451949 +0000 UTC m=+1069.220562068" Nov 04 01:16:27.931356 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:27.931321 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:27.932804 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:27.932780 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-tqg84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body= Nov 04 01:16:27.932951 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:27.932832 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" podUID="c13769b4-f82d-460e-aa0f-0e882b40346c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" Nov 04 01:16:28.931421 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:28.931388 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-tqg84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body= Nov 04 01:16:28.931830 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:28.931439 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" podUID="c13769b4-f82d-460e-aa0f-0e882b40346c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" Nov 04 01:16:29.889704 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:16:29.889667 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:16:29.931602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:29.931544 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-tqg84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body= Nov 04 01:16:29.931982 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:29.931617 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" podUID="c13769b4-f82d-460e-aa0f-0e882b40346c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" Nov 04 01:16:30.471193 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:30.471146 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:16:30.931770 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:30.931736 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-tqg84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body= Nov 04 01:16:30.932239 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:30.931825 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" podUID="c13769b4-f82d-460e-aa0f-0e882b40346c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" Nov 04 01:16:31.931194 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:31.931153 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-tqg84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body= Nov 04 01:16:31.931393 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:31.931212 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" podUID="c13769b4-f82d-460e-aa0f-0e882b40346c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" Nov 04 01:16:32.077336 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:32.077300 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:16:32.449989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:32.449956 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:16:33.115831 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:33.115798 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:33.116255 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:33.116149 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:33.116898 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:33.116878 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" Nov 04 01:16:33.533908 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:33.533867 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:16:34.063532 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:34.063500 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:16:35.065592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:35.064442 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:16:35.635905 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:35.635871 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:16:36.784696 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:36.784663 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:16:39.305453 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:39.305348 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:16:39.914480 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:39.914449 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:16:39.942790 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:39.942762 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:16:40.620322 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.620289 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt"] Nov 04 01:16:40.625055 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.625032 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.637070 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.637051 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt"] Nov 04 01:16:40.682941 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.682905 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a232696a-6a25-4803-a2a0-a9888279273c-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.683083 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.682954 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a232696a-6a25-4803-a2a0-a9888279273c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.683083 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.682984 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a232696a-6a25-4803-a2a0-a9888279273c-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.683237 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.683104 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzj82\" (UniqueName: \"kubernetes.io/projected/a232696a-6a25-4803-a2a0-a9888279273c-kube-api-access-rzj82\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.683237 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.683194 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a232696a-6a25-4803-a2a0-a9888279273c-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.683308 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.683252 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a232696a-6a25-4803-a2a0-a9888279273c-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.683357 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.683332 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a232696a-6a25-4803-a2a0-a9888279273c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.683427 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.683408 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a232696a-6a25-4803-a2a0-a9888279273c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.683482 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.683447 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a232696a-6a25-4803-a2a0-a9888279273c-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.784802 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.784761 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a232696a-6a25-4803-a2a0-a9888279273c-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.784987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.784824 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a232696a-6a25-4803-a2a0-a9888279273c-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.785068 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.784976 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a232696a-6a25-4803-a2a0-a9888279273c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.785068 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.785052 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a232696a-6a25-4803-a2a0-a9888279273c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.785278 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.785258 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a232696a-6a25-4803-a2a0-a9888279273c-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.785460 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.785171 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a232696a-6a25-4803-a2a0-a9888279273c-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.785789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.785517 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a232696a-6a25-4803-a2a0-a9888279273c-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.785789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.785532 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a232696a-6a25-4803-a2a0-a9888279273c-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.785789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.785583 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a232696a-6a25-4803-a2a0-a9888279273c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.785789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.785628 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a232696a-6a25-4803-a2a0-a9888279273c-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.785789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.785670 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzj82\" (UniqueName: \"kubernetes.io/projected/a232696a-6a25-4803-a2a0-a9888279273c-kube-api-access-rzj82\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.785789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.785842 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a232696a-6a25-4803-a2a0-a9888279273c-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.786242 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.786127 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a232696a-6a25-4803-a2a0-a9888279273c-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.786334 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.786307 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a232696a-6a25-4803-a2a0-a9888279273c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.787738 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.787715 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a232696a-6a25-4803-a2a0-a9888279273c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.788555 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.788533 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a232696a-6a25-4803-a2a0-a9888279273c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.805072 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.805045 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a232696a-6a25-4803-a2a0-a9888279273c-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.805542 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.805516 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzj82\" (UniqueName: \"kubernetes.io/projected/a232696a-6a25-4803-a2a0-a9888279273c-kube-api-access-rzj82\") pod \"maas-default-gateway-openshift-default-75d7c785f-lkkwt\" (UID: \"a232696a-6a25-4803-a2a0-a9888279273c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:40.945071 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:40.944966 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:41.282947 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:41.282912 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt"] Nov 04 01:16:41.300595 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:16:41.296887 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda232696a_6a25_4803_a2a0_a9888279273c.slice/crio-6fb376300ae94b39bff0c03bc5cd280a0d01d831859e50b1fb8c072e1f7443f8 WatchSource:0}: Error finding container 6fb376300ae94b39bff0c03bc5cd280a0d01d831859e50b1fb8c072e1f7443f8: Status 404 returned error can't find the container with id 6fb376300ae94b39bff0c03bc5cd280a0d01d831859e50b1fb8c072e1f7443f8 Nov 04 01:16:41.304907 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:41.304850 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:16:41.305203 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:41.305160 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:16:41.305448 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:41.305417 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:16:41.815454 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:41.815405 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:16:42.155221 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:42.155137 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" event={"ID":"a232696a-6a25-4803-a2a0-a9888279273c","Type":"ContainerStarted","Data":"80f5ccc133b4a1043d4f794184162ad97f31a8bf5774465cce101ffd83360c8a"} Nov 04 01:16:42.155221 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:42.155189 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" event={"ID":"a232696a-6a25-4803-a2a0-a9888279273c","Type":"ContainerStarted","Data":"6fb376300ae94b39bff0c03bc5cd280a0d01d831859e50b1fb8c072e1f7443f8"} Nov 04 01:16:42.190507 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:42.190428 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" podStartSLOduration=2.190407004 podStartE2EDuration="2.190407004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:42.184203239 +0000 UTC m=+1084.299313358" watchObservedRunningTime="2026-11-04 01:16:42.190407004 +0000 UTC m=+1084.305517124" Nov 04 01:16:42.646984 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:42.646914 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:16:42.945929 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:42.945847 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:42.947330 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:42.947305 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lkkwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Nov 04 01:16:42.947476 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:42.947361 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" podUID="a232696a-6a25-4803-a2a0-a9888279273c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Nov 04 01:16:43.946356 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:43.946323 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lkkwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Nov 04 01:16:43.946974 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:43.946385 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" podUID="a232696a-6a25-4803-a2a0-a9888279273c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Nov 04 01:16:44.947014 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:44.946971 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lkkwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Nov 04 01:16:44.947648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:44.947603 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" podUID="a232696a-6a25-4803-a2a0-a9888279273c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Nov 04 01:16:45.589657 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:45.589618 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:16:45.946324 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:45.946236 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lkkwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Nov 04 01:16:45.946324 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:45.946299 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" podUID="a232696a-6a25-4803-a2a0-a9888279273c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Nov 04 01:16:46.946192 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:46.946158 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lkkwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Nov 04 01:16:46.946754 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:46.946234 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" podUID="a232696a-6a25-4803-a2a0-a9888279273c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Nov 04 01:16:47.945618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:47.945549 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lkkwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Nov 04 01:16:47.945800 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:47.945642 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" podUID="a232696a-6a25-4803-a2a0-a9888279273c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Nov 04 01:16:49.945581 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:49.945528 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lkkwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:16:49.946022 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:49.945619 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" podUID="a232696a-6a25-4803-a2a0-a9888279273c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:50.986789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:50.986746 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lkkwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:16:50.987284 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:50.986820 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" podUID="a232696a-6a25-4803-a2a0-a9888279273c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:51.620221 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:51.620188 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:51.620722 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:51.620686 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:51.621552 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:51.621531 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" Nov 04 01:16:58.535064 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:58.535023 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:16:58.717013 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:16:58.716979 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:16:59.865126 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:16:59.865094 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:17:03.748730 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:17:03.748699 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:17:03.880333 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:17:03.880297 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:17:21.358538 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:17:21.358442 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:17:21.493764 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:17:21.493734 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:17:29.864939 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:17:29.864905 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:17:30.016366 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:17:30.016338 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:17:30.250052 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:17:30.249982 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:17:30.522930 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:17:30.522899 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:17:31.841309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:17:31.841277 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:17:32.000017 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:17:31.999990 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:17:33.405434 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:17:33.405404 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:17:33.559230 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:17:33.559200 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:17:56.389004 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:17:56.388968 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:17:56.534053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:17:56.534023 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:17:56.941187 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:17:56.941152 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:17:57.160481 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:17:57.160451 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:17:57.320611 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:17:57.320557 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:17:57.465733 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:17:57.465695 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:17:57.624851 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:17:57.624755 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:17:57.785625 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:17:57.785590 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:17:59.865823 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:17:59.865788 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:18:00.701094 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:00.701056 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:01.037792 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:01.037731 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:01.259917 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:01.259884 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:01.429582 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:01.429487 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:01.860480 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:01.860440 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:02.041455 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:02.041421 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:02.205150 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:02.205071 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:02.365279 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:02.365239 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:02.700988 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:02.700951 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:02.866189 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:02.866157 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:03.149591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:03.149537 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:03.566534 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:03.566491 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:03.726064 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:03.726021 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:04.017074 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:04.017042 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:04.159888 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:04.159827 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:04.673073 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:04.673031 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:04.827775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:04.827738 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:05.283727 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.283693 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:05.342854 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.342820 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd"] Nov 04 01:18:05.346048 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.346031 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd" Nov 04 01:18:05.355914 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.355887 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd"] Nov 04 01:18:05.411957 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.411917 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-ee876a-dockercfg-9kjqt\"" Nov 04 01:18:05.412100 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.412044 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-ee876a\"" Nov 04 01:18:05.437728 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.437468 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38bc84cb-29d4-4979-8d64-17393f82c169-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd\" (UID: \"38bc84cb-29d4-4979-8d64-17393f82c169\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd" Nov 04 01:18:05.437728 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.437521 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffgtx\" (UniqueName: \"kubernetes.io/projected/38bc84cb-29d4-4979-8d64-17393f82c169-kube-api-access-ffgtx\") pod \"payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd\" (UID: \"38bc84cb-29d4-4979-8d64-17393f82c169\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd" Nov 04 01:18:05.438077 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.438046 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:05.538591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.538492 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38bc84cb-29d4-4979-8d64-17393f82c169-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd\" (UID: \"38bc84cb-29d4-4979-8d64-17393f82c169\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd" Nov 04 01:18:05.538591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.538535 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffgtx\" (UniqueName: \"kubernetes.io/projected/38bc84cb-29d4-4979-8d64-17393f82c169-kube-api-access-ffgtx\") pod \"payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd\" (UID: \"38bc84cb-29d4-4979-8d64-17393f82c169\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd" Nov 04 01:18:05.539304 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.539263 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38bc84cb-29d4-4979-8d64-17393f82c169-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd\" (UID: \"38bc84cb-29d4-4979-8d64-17393f82c169\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd" Nov 04 01:18:05.552853 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.552831 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffgtx\" (UniqueName: \"kubernetes.io/projected/38bc84cb-29d4-4979-8d64-17393f82c169-kube-api-access-ffgtx\") pod \"payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd\" (UID: \"38bc84cb-29d4-4979-8d64-17393f82c169\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd" Nov 04 01:18:05.598954 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.598928 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:05.637739 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.637714 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6"] Nov 04 01:18:05.641686 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.641667 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6" Nov 04 01:18:05.649237 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.648632 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-586541\"" Nov 04 01:18:05.651357 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.650794 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6"] Nov 04 01:18:05.659026 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.659007 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd" Nov 04 01:18:05.740332 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.740306 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngqmt\" (UniqueName: \"kubernetes.io/projected/456612ef-2ef3-4b16-bab3-1277c84311c5-kube-api-access-ngqmt\") pod \"payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6\" (UID: \"456612ef-2ef3-4b16-bab3-1277c84311c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6" Nov 04 01:18:05.740478 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.740367 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/456612ef-2ef3-4b16-bab3-1277c84311c5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6\" (UID: \"456612ef-2ef3-4b16-bab3-1277c84311c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6" Nov 04 01:18:05.742897 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.742871 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:05.843794 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.842048 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/456612ef-2ef3-4b16-bab3-1277c84311c5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6\" (UID: \"456612ef-2ef3-4b16-bab3-1277c84311c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6" Nov 04 01:18:05.843794 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.842221 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngqmt\" (UniqueName: \"kubernetes.io/projected/456612ef-2ef3-4b16-bab3-1277c84311c5-kube-api-access-ngqmt\") pod \"payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6\" (UID: \"456612ef-2ef3-4b16-bab3-1277c84311c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6" Nov 04 01:18:05.850189 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.849591 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/456612ef-2ef3-4b16-bab3-1277c84311c5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6\" (UID: \"456612ef-2ef3-4b16-bab3-1277c84311c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6" Nov 04 01:18:05.852607 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.852552 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngqmt\" (UniqueName: \"kubernetes.io/projected/456612ef-2ef3-4b16-bab3-1277c84311c5-kube-api-access-ngqmt\") pod \"payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6\" (UID: \"456612ef-2ef3-4b16-bab3-1277c84311c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6" Nov 04 01:18:05.937823 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.937794 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd"] Nov 04 01:18:05.941260 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:18:05.941231 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38bc84cb_29d4_4979_8d64_17393f82c169.slice/crio-a984f3bf7778a44ff31b9f8afd38d20c3618f817b796384cd9e9256683d3861f WatchSource:0}: Error finding container a984f3bf7778a44ff31b9f8afd38d20c3618f817b796384cd9e9256683d3861f: Status 404 returned error can't find the container with id a984f3bf7778a44ff31b9f8afd38d20c3618f817b796384cd9e9256683d3861f Nov 04 01:18:05.959187 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:05.959165 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6" Nov 04 01:18:06.025335 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:06.025308 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:18:06.177131 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:06.177053 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:06.234784 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:06.234756 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6"] Nov 04 01:18:06.237606 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:18:06.237579 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456612ef_2ef3_4b16_bab3_1277c84311c5.slice/crio-64f8cb80a5761a9ea8287603f47ddd233cc69ebd86c36478cb6f7890f52d803e WatchSource:0}: Error finding container 64f8cb80a5761a9ea8287603f47ddd233cc69ebd86c36478cb6f7890f52d803e: Status 404 returned error can't find the container with id 64f8cb80a5761a9ea8287603f47ddd233cc69ebd86c36478cb6f7890f52d803e Nov 04 01:18:06.317298 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:06.317275 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:06.549273 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:06.549239 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd" event={"ID":"38bc84cb-29d4-4979-8d64-17393f82c169","Type":"ContainerStarted","Data":"a984f3bf7778a44ff31b9f8afd38d20c3618f817b796384cd9e9256683d3861f"} Nov 04 01:18:06.550601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:06.550547 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6" event={"ID":"456612ef-2ef3-4b16-bab3-1277c84311c5","Type":"ContainerStarted","Data":"64f8cb80a5761a9ea8287603f47ddd233cc69ebd86c36478cb6f7890f52d803e"} Nov 04 01:18:07.032869 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:07.032762 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:07.814002 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:07.813930 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8"] Nov 04 01:18:07.817978 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:07.817950 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" Nov 04 01:18:07.828930 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:07.828900 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-ee876a-serving-cert\"" Nov 04 01:18:07.830580 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:07.830479 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8"] Nov 04 01:18:07.917515 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:07.917480 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:07.959882 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:07.959840 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5744bca-e67c-4284-b08e-1a3cc950ca77-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8\" (UID: \"f5744bca-e67c-4284-b08e-1a3cc950ca77\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" Nov 04 01:18:07.959882 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:07.959887 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5744bca-e67c-4284-b08e-1a3cc950ca77-metrics-certs\") pod \"maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8\" (UID: \"f5744bca-e67c-4284-b08e-1a3cc950ca77\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" Nov 04 01:18:07.960133 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:07.959923 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5744bca-e67c-4284-b08e-1a3cc950ca77-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8\" (UID: \"f5744bca-e67c-4284-b08e-1a3cc950ca77\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" Nov 04 01:18:07.960133 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:07.959973 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zppgx\" (UniqueName: \"kubernetes.io/projected/f5744bca-e67c-4284-b08e-1a3cc950ca77-kube-api-access-zppgx\") pod \"maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8\" (UID: \"f5744bca-e67c-4284-b08e-1a3cc950ca77\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" Nov 04 01:18:08.061261 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:08.061223 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5744bca-e67c-4284-b08e-1a3cc950ca77-metrics-certs\") pod \"maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8\" (UID: \"f5744bca-e67c-4284-b08e-1a3cc950ca77\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" Nov 04 01:18:08.061648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:08.061319 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5744bca-e67c-4284-b08e-1a3cc950ca77-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8\" (UID: \"f5744bca-e67c-4284-b08e-1a3cc950ca77\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" Nov 04 01:18:08.062340 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:08.061654 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zppgx\" (UniqueName: \"kubernetes.io/projected/f5744bca-e67c-4284-b08e-1a3cc950ca77-kube-api-access-zppgx\") pod \"maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8\" (UID: \"f5744bca-e67c-4284-b08e-1a3cc950ca77\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" Nov 04 01:18:08.062340 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:08.061745 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5744bca-e67c-4284-b08e-1a3cc950ca77-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8\" (UID: \"f5744bca-e67c-4284-b08e-1a3cc950ca77\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" Nov 04 01:18:08.074989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:08.072092 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:08.074989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:08.072649 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5744bca-e67c-4284-b08e-1a3cc950ca77-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8\" (UID: \"f5744bca-e67c-4284-b08e-1a3cc950ca77\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" Nov 04 01:18:08.076279 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:08.076212 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5744bca-e67c-4284-b08e-1a3cc950ca77-metrics-certs\") pod \"maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8\" (UID: \"f5744bca-e67c-4284-b08e-1a3cc950ca77\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" Nov 04 01:18:08.076279 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:08.076212 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5744bca-e67c-4284-b08e-1a3cc950ca77-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8\" (UID: \"f5744bca-e67c-4284-b08e-1a3cc950ca77\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" Nov 04 01:18:08.080446 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:08.080421 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zppgx\" (UniqueName: \"kubernetes.io/projected/f5744bca-e67c-4284-b08e-1a3cc950ca77-kube-api-access-zppgx\") pod \"maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8\" (UID: \"f5744bca-e67c-4284-b08e-1a3cc950ca77\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" Nov 04 01:18:08.154076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:08.154036 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" Nov 04 01:18:08.473152 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:08.473078 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8"] Nov 04 01:18:08.475816 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:18:08.475783 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5744bca_e67c_4284_b08e_1a3cc950ca77.slice/crio-6d7ce0b886111d41798f2d59ce2b5f82257b88b4ed337de053c331a5e787bc9c WatchSource:0}: Error finding container 6d7ce0b886111d41798f2d59ce2b5f82257b88b4ed337de053c331a5e787bc9c: Status 404 returned error can't find the container with id 6d7ce0b886111d41798f2d59ce2b5f82257b88b4ed337de053c331a5e787bc9c Nov 04 01:18:08.563799 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:08.563759 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" event={"ID":"f5744bca-e67c-4284-b08e-1a3cc950ca77","Type":"ContainerStarted","Data":"6d7ce0b886111d41798f2d59ce2b5f82257b88b4ed337de053c331a5e787bc9c"} Nov 04 01:18:08.702699 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:08.702667 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:08.852370 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:08.852331 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:09.498469 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.498445 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6"] Nov 04 01:18:09.501797 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.501777 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" Nov 04 01:18:09.509080 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.509059 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-9c8bf5-serving-cert\"" Nov 04 01:18:09.514617 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.513588 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6"] Nov 04 01:18:09.570175 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.570140 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6" event={"ID":"456612ef-2ef3-4b16-bab3-1277c84311c5","Type":"ContainerStarted","Data":"83940cffbec9e8adf33b391cef72116ec984dc3d29e1a93574d6bbe69d11595b"} Nov 04 01:18:09.570326 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.570259 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6" Nov 04 01:18:09.571644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.571623 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd" event={"ID":"38bc84cb-29d4-4979-8d64-17393f82c169","Type":"ContainerStarted","Data":"ad53f09f2ee93b18a9a3a3deaee7b34c34235f63b5903d787886e5373a3191ad"} Nov 04 01:18:09.571728 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.571670 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd" Nov 04 01:18:09.573101 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.573080 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" event={"ID":"f5744bca-e67c-4284-b08e-1a3cc950ca77","Type":"ContainerStarted","Data":"50e44dfd0f5152526674b042e13e7b01766526e496437eaf6612e993a616956b"} Nov 04 01:18:09.573268 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.573252 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" Nov 04 01:18:09.589576 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.589531 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6" podStartSLOduration=1.443774775 podStartE2EDuration="4.589519418s" totalImagesPullingTime="3.145744643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:09.587953616 +0000 UTC m=+1171.703063739" watchObservedRunningTime="2026-11-04 01:18:09.589519418 +0000 UTC m=+1171.704629530" Nov 04 01:18:09.605491 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.605457 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd" podStartSLOduration=1.5717312350000001 podStartE2EDuration="4.605445826s" totalImagesPullingTime="3.033714591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:09.604526954 +0000 UTC m=+1171.719637075" watchObservedRunningTime="2026-11-04 01:18:09.605445826 +0000 UTC m=+1171.720555941" Nov 04 01:18:09.608413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.608394 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:09.622845 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.622807 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" podStartSLOduration=2.169507512 podStartE2EDuration="2.622797968s" totalImagesPullingTime="453.290456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:09.62125978 +0000 UTC m=+1171.736369895" watchObservedRunningTime="2026-11-04 01:18:09.622797968 +0000 UTC m=+1171.737908151" Nov 04 01:18:09.678135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.678081 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhwwl\" (UniqueName: \"kubernetes.io/projected/9961d5eb-fd3c-4651-96a8-861577e1ad21-kube-api-access-xhwwl\") pod \"maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6\" (UID: \"9961d5eb-fd3c-4651-96a8-861577e1ad21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" Nov 04 01:18:09.678220 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.678204 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9961d5eb-fd3c-4651-96a8-861577e1ad21-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6\" (UID: \"9961d5eb-fd3c-4651-96a8-861577e1ad21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" Nov 04 01:18:09.678475 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.678399 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9961d5eb-fd3c-4651-96a8-861577e1ad21-maas-api-tls\") pod \"maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6\" (UID: \"9961d5eb-fd3c-4651-96a8-861577e1ad21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" Nov 04 01:18:09.678475 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.678437 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9961d5eb-fd3c-4651-96a8-861577e1ad21-metrics-certs\") pod \"maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6\" (UID: \"9961d5eb-fd3c-4651-96a8-861577e1ad21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" Nov 04 01:18:09.755210 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.755173 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:09.781053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.780015 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9961d5eb-fd3c-4651-96a8-861577e1ad21-maas-api-tls\") pod \"maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6\" (UID: \"9961d5eb-fd3c-4651-96a8-861577e1ad21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" Nov 04 01:18:09.781053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.780064 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9961d5eb-fd3c-4651-96a8-861577e1ad21-metrics-certs\") pod \"maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6\" (UID: \"9961d5eb-fd3c-4651-96a8-861577e1ad21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" Nov 04 01:18:09.781053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.780122 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhwwl\" (UniqueName: \"kubernetes.io/projected/9961d5eb-fd3c-4651-96a8-861577e1ad21-kube-api-access-xhwwl\") pod \"maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6\" (UID: \"9961d5eb-fd3c-4651-96a8-861577e1ad21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" Nov 04 01:18:09.781053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.780211 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9961d5eb-fd3c-4651-96a8-861577e1ad21-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6\" (UID: \"9961d5eb-fd3c-4651-96a8-861577e1ad21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" Nov 04 01:18:09.781053 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:18:09.780283 4852 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-serving-cert: secret "maas-api-e2e-worker-w5-9c8bf5-serving-cert" not found Nov 04 01:18:09.781053 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:18:09.780373 4852 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9961d5eb-fd3c-4651-96a8-861577e1ad21-maas-api-tls podName:9961d5eb-fd3c-4651-96a8-861577e1ad21 nodeName:}" failed. No retries permitted until 2026-11-04 01:18:10.28034955 +0000 UTC m=+1172.395459669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9961d5eb-fd3c-4651-96a8-861577e1ad21-maas-api-tls") pod "maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" (UID: "9961d5eb-fd3c-4651-96a8-861577e1ad21") : secret "maas-api-e2e-worker-w5-9c8bf5-serving-cert" not found Nov 04 01:18:09.781053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.781007 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9961d5eb-fd3c-4651-96a8-861577e1ad21-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6\" (UID: \"9961d5eb-fd3c-4651-96a8-861577e1ad21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" Nov 04 01:18:09.783843 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.783779 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9961d5eb-fd3c-4651-96a8-861577e1ad21-metrics-certs\") pod \"maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6\" (UID: \"9961d5eb-fd3c-4651-96a8-861577e1ad21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" Nov 04 01:18:09.790871 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:09.790652 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhwwl\" (UniqueName: \"kubernetes.io/projected/9961d5eb-fd3c-4651-96a8-861577e1ad21-kube-api-access-xhwwl\") pod \"maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6\" (UID: \"9961d5eb-fd3c-4651-96a8-861577e1ad21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" Nov 04 01:18:10.047607 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:10.047523 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:10.285357 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:10.285325 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9961d5eb-fd3c-4651-96a8-861577e1ad21-maas-api-tls\") pod \"maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6\" (UID: \"9961d5eb-fd3c-4651-96a8-861577e1ad21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" Nov 04 01:18:10.287851 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:10.287830 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9961d5eb-fd3c-4651-96a8-861577e1ad21-maas-api-tls\") pod \"maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6\" (UID: \"9961d5eb-fd3c-4651-96a8-861577e1ad21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" Nov 04 01:18:10.419201 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:10.418763 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" Nov 04 01:18:10.430084 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:10.428982 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:10.744248 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:10.744213 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6"] Nov 04 01:18:11.282000 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:11.281956 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:11.584320 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:11.584287 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" event={"ID":"9961d5eb-fd3c-4651-96a8-861577e1ad21","Type":"ContainerStarted","Data":"e366d7e972425458663f934dfc1bfa8bbca855ddd3e2561cd0f222a78ea8fe36"} Nov 04 01:18:11.584495 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:11.584343 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" event={"ID":"9961d5eb-fd3c-4651-96a8-861577e1ad21","Type":"ContainerStarted","Data":"7843e0d21f30e3b8590cc65a9eea52c4588ba4445699901cf4b5743897793071"} Nov 04 01:18:11.584495 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:11.584406 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" Nov 04 01:18:11.603236 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:11.603186 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" podStartSLOduration=2.136560069 podStartE2EDuration="2.603168599s" totalImagesPullingTime="466.60853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:11.600675323 +0000 UTC m=+1173.715785468" watchObservedRunningTime="2026-11-04 01:18:11.603168599 +0000 UTC m=+1173.718278717" Nov 04 01:18:12.212771 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:12.212733 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:13.033630 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:13.033590 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:13.880686 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:13.880646 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:14.628073 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:14.628029 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:15.326347 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:15.326314 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:15.583873 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:15.583798 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" Nov 04 01:18:15.722293 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:15.722260 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:15.863515 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:15.863435 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:16.123913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:16.123818 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:16.423723 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:16.423641 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:16.570445 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:16.570417 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:16.738597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:16.738487 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:16.962635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:16.960033 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:17.594268 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:17.594241 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" Nov 04 01:18:17.798660 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:17.798630 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:18.552233 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.552205 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb"] Nov 04 01:18:18.555669 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.555649 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:18.559790 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.559771 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d7e45d\"/\"kube-root-ca.crt\"" Nov 04 01:18:18.562652 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.562623 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d7e45d\"/\"openshift-service-ca.crt\"" Nov 04 01:18:18.564064 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.564041 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d7e45d\"/\"simulator-w2-d7e45d-kserve-self-signed-certs\"" Nov 04 01:18:18.564539 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.564518 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d7e45d\"/\"default-dockercfg-9sgp7\"" Nov 04 01:18:18.566279 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.566256 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb"] Nov 04 01:18:18.573263 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.573235 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-model-cache\") pod \"simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:18.573406 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.573303 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zt8x\" (UniqueName: \"kubernetes.io/projected/ecc4ef25-c12e-4b56-b505-1435c1e0822c-kube-api-access-6zt8x\") pod \"simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:18.573406 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.573382 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-dshm\") pod \"simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:18.573521 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.573416 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-tmp-dir\") pod \"simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:18.573521 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.573443 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-home\") pod \"simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:18.573521 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.573494 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc4ef25-c12e-4b56-b505-1435c1e0822c-tls-certs\") pod \"simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:18.621517 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.621486 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:18.674436 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.674410 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc4ef25-c12e-4b56-b505-1435c1e0822c-tls-certs\") pod \"simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:18.674623 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.674464 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-model-cache\") pod \"simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:18.674623 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.674497 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zt8x\" (UniqueName: \"kubernetes.io/projected/ecc4ef25-c12e-4b56-b505-1435c1e0822c-kube-api-access-6zt8x\") pod \"simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:18.674767 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.674748 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-dshm\") pod \"simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:18.674832 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.674778 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-tmp-dir\") pod \"simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:18.674832 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.674800 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-home\") pod \"simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:18.675009 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.674990 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-model-cache\") pod \"simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:18.675110 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.675090 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-tmp-dir\") pod \"simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:18.675181 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.675164 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-home\") pod \"simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:18.677072 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.677047 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-dshm\") pod \"simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:18.677229 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.677213 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc4ef25-c12e-4b56-b505-1435c1e0822c-tls-certs\") pod \"simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:18.682309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.682289 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zt8x\" (UniqueName: \"kubernetes.io/projected/ecc4ef25-c12e-4b56-b505-1435c1e0822c-kube-api-access-6zt8x\") pod \"simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:18.737644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.737619 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:18:18.838477 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.838337 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:18.869715 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:18.869686 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:19.153656 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.153268 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb"] Nov 04 01:18:19.165799 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:18:19.165731 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecc4ef25_c12e_4b56_b505_1435c1e0822c.slice/crio-005424728e7b934d24e694c55f6c21d14432b523cac734489fd2b17ff6951a04 WatchSource:0}: Error finding container 005424728e7b934d24e694c55f6c21d14432b523cac734489fd2b17ff6951a04: Status 404 returned error can't find the container with id 005424728e7b934d24e694c55f6c21d14432b523cac734489fd2b17ff6951a04 Nov 04 01:18:19.221150 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.221114 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms"] Nov 04 01:18:19.226195 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.226171 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:19.238406 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.238382 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-099dd9\"/\"openshift-service-ca.crt\"" Nov 04 01:18:19.238592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.238437 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-099dd9\"/\"default-dockercfg-jnlg9\"" Nov 04 01:18:19.238592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.238475 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-099dd9\"/\"kube-root-ca.crt\"" Nov 04 01:18:19.239813 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.238726 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-099dd9\"/\"simulator-w4-099dd9-kserve-self-signed-certs\"" Nov 04 01:18:19.239813 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.239734 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms"] Nov 04 01:18:19.280758 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.280728 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c69e53e0-661a-4a14-94de-d4450a554c80-tls-certs\") pod \"simulator-w4-099dd9-kserve-5fb6586d44-rpkms\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:19.280970 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.280782 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-model-cache\") pod \"simulator-w4-099dd9-kserve-5fb6586d44-rpkms\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:19.280970 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.280913 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-home\") pod \"simulator-w4-099dd9-kserve-5fb6586d44-rpkms\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:19.281071 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.280975 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-tmp-dir\") pod \"simulator-w4-099dd9-kserve-5fb6586d44-rpkms\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:19.281071 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.281050 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgpfr\" (UniqueName: \"kubernetes.io/projected/c69e53e0-661a-4a14-94de-d4450a554c80-kube-api-access-pgpfr\") pod \"simulator-w4-099dd9-kserve-5fb6586d44-rpkms\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:19.281178 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.281111 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-dshm\") pod \"simulator-w4-099dd9-kserve-5fb6586d44-rpkms\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:19.382109 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.382079 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-home\") pod \"simulator-w4-099dd9-kserve-5fb6586d44-rpkms\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:19.382265 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.382129 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-tmp-dir\") pod \"simulator-w4-099dd9-kserve-5fb6586d44-rpkms\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:19.382265 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.382176 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgpfr\" (UniqueName: \"kubernetes.io/projected/c69e53e0-661a-4a14-94de-d4450a554c80-kube-api-access-pgpfr\") pod \"simulator-w4-099dd9-kserve-5fb6586d44-rpkms\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:19.382265 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.382213 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-dshm\") pod \"simulator-w4-099dd9-kserve-5fb6586d44-rpkms\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:19.382412 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.382268 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c69e53e0-661a-4a14-94de-d4450a554c80-tls-certs\") pod \"simulator-w4-099dd9-kserve-5fb6586d44-rpkms\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:19.382412 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.382307 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-model-cache\") pod \"simulator-w4-099dd9-kserve-5fb6586d44-rpkms\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:19.382607 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.382585 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-home\") pod \"simulator-w4-099dd9-kserve-5fb6586d44-rpkms\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:19.382683 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.382618 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-tmp-dir\") pod \"simulator-w4-099dd9-kserve-5fb6586d44-rpkms\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:19.382683 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.382660 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-model-cache\") pod \"simulator-w4-099dd9-kserve-5fb6586d44-rpkms\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:19.384601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.384581 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-dshm\") pod \"simulator-w4-099dd9-kserve-5fb6586d44-rpkms\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:19.384862 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.384848 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c69e53e0-661a-4a14-94de-d4450a554c80-tls-certs\") pod \"simulator-w4-099dd9-kserve-5fb6586d44-rpkms\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:19.401695 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.401676 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgpfr\" (UniqueName: \"kubernetes.io/projected/c69e53e0-661a-4a14-94de-d4450a554c80-kube-api-access-pgpfr\") pod \"simulator-w4-099dd9-kserve-5fb6586d44-rpkms\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:19.475876 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.475801 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:19.549671 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.549638 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:19.633294 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.630371 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" event={"ID":"ecc4ef25-c12e-4b56-b505-1435c1e0822c","Type":"ContainerStarted","Data":"005424728e7b934d24e694c55f6c21d14432b523cac734489fd2b17ff6951a04"} Nov 04 01:18:19.715275 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.715175 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:19.850212 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:19.850157 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms"] Nov 04 01:18:20.441247 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:20.441155 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:20.582017 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:20.581977 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6" Nov 04 01:18:20.582362 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:20.582342 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd" Nov 04 01:18:20.640361 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:20.640332 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" event={"ID":"c69e53e0-661a-4a14-94de-d4450a554c80","Type":"ContainerStarted","Data":"969c47d7ec553c872c82db6103061d0b136673d218ace32abebe1d66013c411f"} Nov 04 01:18:20.931600 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:20.930942 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:21.069588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:21.069498 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:21.219779 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:21.219729 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:21.620269 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:21.620237 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:21.646132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:21.646094 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" event={"ID":"c69e53e0-661a-4a14-94de-d4450a554c80","Type":"ContainerStarted","Data":"a5e4847f710c596a6473fff78d95944d7d4a81e35b0ea084d344b6e6830ff388"} Nov 04 01:18:21.647752 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:21.647725 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" event={"ID":"ecc4ef25-c12e-4b56-b505-1435c1e0822c","Type":"ContainerStarted","Data":"0f1b55ff68e6a9d0c515e53b44b1636b6a5ece6920f3f61d39da27f1924d43e8"} Nov 04 01:18:21.685466 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:21.685419 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podStartSLOduration=1.769717424 podStartE2EDuration="2.685405014s" totalImagesPullingTime="915.68759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:21.681669803 +0000 UTC m=+1183.796779921" watchObservedRunningTime="2026-11-04 01:18:21.685405014 +0000 UTC m=+1183.800515138" Nov 04 01:18:21.720384 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:21.720328 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podStartSLOduration=2.301318977 podStartE2EDuration="3.720310366s" totalImagesPullingTime="1.418991389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:21.720039229 +0000 UTC m=+1183.835149349" watchObservedRunningTime="2026-11-04 01:18:21.720310366 +0000 UTC m=+1183.835420528" Nov 04 01:18:21.870034 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:21.869990 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:21.924184 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:21.924100 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:22.389755 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:22.389721 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:22.550656 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:22.550623 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:23.351547 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:23.351521 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:24.282695 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:24.282664 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:24.930768 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:24.930725 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:25.732403 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:25.732368 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:26.091373 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:26.091342 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:26.446677 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:26.446590 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:26.723367 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:26.723292 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:26.891659 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:26.891619 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:27.221617 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:27.221583 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:27.523337 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:27.523301 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:27.735717 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:27.735680 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:27.943905 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:27.943815 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:28.250625 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:28.250529 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:28.660615 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:28.660574 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:28.870122 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:28.870090 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:28.883020 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:28.882995 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:29.489330 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:29.489297 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:29.550288 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:29.550258 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:29.563000 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:29.562970 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:29.692856 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:29.692828 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:18:29.693690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:29.693673 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:18:29.866095 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:18:29.865992 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:18:30.115056 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.114363 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:30.558031 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.558001 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:30.558808 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.558787 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94"] Nov 04 01:18:30.563412 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.563391 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:30.564398 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.564374 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94"] Nov 04 01:18:30.569244 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.569223 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ca55c1\"/\"openshift-service-ca.crt\"" Nov 04 01:18:30.570195 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.570175 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ca55c1\"/\"kube-root-ca.crt\"" Nov 04 01:18:30.570313 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.570237 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ca55c1\"/\"default-dockercfg-zwvp4\"" Nov 04 01:18:30.570313 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.570237 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ca55c1\"/\"simulator-w3-ca55c1-kserve-self-signed-certs\"" Nov 04 01:18:30.582443 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.582420 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-home\") pod \"simulator-w3-ca55c1-kserve-588fc66fb-6ql94\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:30.582606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.582466 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-dshm\") pod \"simulator-w3-ca55c1-kserve-588fc66fb-6ql94\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:30.582606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.582492 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-model-cache\") pod \"simulator-w3-ca55c1-kserve-588fc66fb-6ql94\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:30.582737 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.582640 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-tmp-dir\") pod \"simulator-w3-ca55c1-kserve-588fc66fb-6ql94\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:30.582737 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.582684 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-tls-certs\") pod \"simulator-w3-ca55c1-kserve-588fc66fb-6ql94\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:30.582838 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.582756 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8wcq\" (UniqueName: \"kubernetes.io/projected/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-kube-api-access-l8wcq\") pod \"simulator-w3-ca55c1-kserve-588fc66fb-6ql94\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:30.683514 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.683474 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-home\") pod \"simulator-w3-ca55c1-kserve-588fc66fb-6ql94\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:30.683703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.683530 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-dshm\") pod \"simulator-w3-ca55c1-kserve-588fc66fb-6ql94\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:30.683703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.683592 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-model-cache\") pod \"simulator-w3-ca55c1-kserve-588fc66fb-6ql94\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:30.683703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.683622 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-tmp-dir\") pod \"simulator-w3-ca55c1-kserve-588fc66fb-6ql94\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:30.683703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.683655 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-tls-certs\") pod \"simulator-w3-ca55c1-kserve-588fc66fb-6ql94\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:30.683877 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.683711 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8wcq\" (UniqueName: \"kubernetes.io/projected/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-kube-api-access-l8wcq\") pod \"simulator-w3-ca55c1-kserve-588fc66fb-6ql94\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:30.684176 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.684153 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-home\") pod \"simulator-w3-ca55c1-kserve-588fc66fb-6ql94\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:30.685789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.684833 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-model-cache\") pod \"simulator-w3-ca55c1-kserve-588fc66fb-6ql94\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:30.685789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.685053 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-tmp-dir\") pod \"simulator-w3-ca55c1-kserve-588fc66fb-6ql94\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:30.687275 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.686980 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-dshm\") pod \"simulator-w3-ca55c1-kserve-588fc66fb-6ql94\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:30.688831 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.688776 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-tls-certs\") pod \"simulator-w3-ca55c1-kserve-588fc66fb-6ql94\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:30.693003 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.692983 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8wcq\" (UniqueName: \"kubernetes.io/projected/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-kube-api-access-l8wcq\") pod \"simulator-w3-ca55c1-kserve-588fc66fb-6ql94\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:30.697918 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.697894 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:30.839667 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.839548 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:30.889247 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:30.889214 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:31.175384 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:31.175358 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94"] Nov 04 01:18:31.517770 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:31.516540 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:31.695816 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:31.695782 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" event={"ID":"c8ff057d-6968-4e4a-b0cf-3c5fbc325173","Type":"ContainerStarted","Data":"8b4cd5bc5d755efafac4fc022f79f3276b2f3335e22191760186e9e2359e1f52"} Nov 04 01:18:31.696176 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:31.695831 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" event={"ID":"c8ff057d-6968-4e4a-b0cf-3c5fbc325173","Type":"ContainerStarted","Data":"2c42bf48c51b8e0d3fe1b9224ec38682398fbc88e3c4767ed4ec7cce2a957a93"} Nov 04 01:18:31.712239 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:31.711978 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podStartSLOduration=1.711960805 podStartE2EDuration="1.711960805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:31.711470064 +0000 UTC m=+1193.826580196" watchObservedRunningTime="2026-11-04 01:18:31.711960805 +0000 UTC m=+1193.827070927" Nov 04 01:18:31.890355 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:31.890271 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:32.319868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:32.319829 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:32.928083 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:32.928048 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:33.456715 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:33.456685 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:18:33.741215 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:33.740952 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:35.277583 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:35.277523 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:36.027148 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:36.027120 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:36.745547 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:36.745509 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:37.274914 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.274884 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s"] Nov 04 01:18:37.279729 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.279697 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:37.286537 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.286503 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ee876a\"/\"openshift-service-ca.crt\"" Nov 04 01:18:37.287517 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.287487 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ee876a\"/\"kube-root-ca.crt\"" Nov 04 01:18:37.289759 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.289734 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s"] Nov 04 01:18:37.323243 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.323222 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ee876a\"/\"default-dockercfg-dhtjx\"" Nov 04 01:18:37.323243 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.323226 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ee876a\"/\"simulator-w0-ee876a-kserve-self-signed-certs\"" Nov 04 01:18:37.346775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.346745 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-model-cache\") pod \"simulator-w0-ee876a-kserve-74549bbf9c-h2g6s\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:37.346775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.346774 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8csqp\" (UniqueName: \"kubernetes.io/projected/abe48140-3cf1-43a8-a814-18f86e309795-kube-api-access-8csqp\") pod \"simulator-w0-ee876a-kserve-74549bbf9c-h2g6s\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:37.346944 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.346803 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-home\") pod \"simulator-w0-ee876a-kserve-74549bbf9c-h2g6s\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:37.346944 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.346887 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-dshm\") pod \"simulator-w0-ee876a-kserve-74549bbf9c-h2g6s\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:37.347035 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.346971 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe48140-3cf1-43a8-a814-18f86e309795-tls-certs\") pod \"simulator-w0-ee876a-kserve-74549bbf9c-h2g6s\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:37.347035 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.347020 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-tmp-dir\") pod \"simulator-w0-ee876a-kserve-74549bbf9c-h2g6s\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:37.394749 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.394716 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:37.448207 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.448171 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-tmp-dir\") pod \"simulator-w0-ee876a-kserve-74549bbf9c-h2g6s\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:37.448389 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.448276 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-model-cache\") pod \"simulator-w0-ee876a-kserve-74549bbf9c-h2g6s\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:37.448389 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.448303 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8csqp\" (UniqueName: \"kubernetes.io/projected/abe48140-3cf1-43a8-a814-18f86e309795-kube-api-access-8csqp\") pod \"simulator-w0-ee876a-kserve-74549bbf9c-h2g6s\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:37.448389 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.448349 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-home\") pod \"simulator-w0-ee876a-kserve-74549bbf9c-h2g6s\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:37.448389 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.448385 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-dshm\") pod \"simulator-w0-ee876a-kserve-74549bbf9c-h2g6s\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:37.448678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.448407 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe48140-3cf1-43a8-a814-18f86e309795-tls-certs\") pod \"simulator-w0-ee876a-kserve-74549bbf9c-h2g6s\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:37.448678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.448621 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-tmp-dir\") pod \"simulator-w0-ee876a-kserve-74549bbf9c-h2g6s\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:37.448678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.448652 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-model-cache\") pod \"simulator-w0-ee876a-kserve-74549bbf9c-h2g6s\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:37.448849 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.448758 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-home\") pod \"simulator-w0-ee876a-kserve-74549bbf9c-h2g6s\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:37.450903 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.450881 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-dshm\") pod \"simulator-w0-ee876a-kserve-74549bbf9c-h2g6s\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:37.451272 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.451249 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe48140-3cf1-43a8-a814-18f86e309795-tls-certs\") pod \"simulator-w0-ee876a-kserve-74549bbf9c-h2g6s\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:37.456891 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.456866 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8csqp\" (UniqueName: \"kubernetes.io/projected/abe48140-3cf1-43a8-a814-18f86e309795-kube-api-access-8csqp\") pod \"simulator-w0-ee876a-kserve-74549bbf9c-h2g6s\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:37.543239 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.541044 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:37.599802 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.599766 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:37.877498 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:37.877425 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s"] Nov 04 01:18:37.880434 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:18:37.880408 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabe48140_3cf1_43a8_a814_18f86e309795.slice/crio-5b6ea630e32e3b2f4abc7538fc12b1972ae696fe5ac056765af0bd85b9b70268 WatchSource:0}: Error finding container 5b6ea630e32e3b2f4abc7538fc12b1972ae696fe5ac056765af0bd85b9b70268: Status 404 returned error can't find the container with id 5b6ea630e32e3b2f4abc7538fc12b1972ae696fe5ac056765af0bd85b9b70268 Nov 04 01:18:38.107038 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:38.107008 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 01:18:38.109466 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:38.109444 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 01:18:38.171645 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:38.171543 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:38.315354 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:38.315325 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:38.731892 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:38.731853 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" event={"ID":"abe48140-3cf1-43a8-a814-18f86e309795","Type":"ContainerStarted","Data":"be820ff88fc70e0765ae7c9f42b07cf26d2db5e006e061512174a0d3263d189a"} Nov 04 01:18:38.731892 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:38.731893 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" event={"ID":"abe48140-3cf1-43a8-a814-18f86e309795","Type":"ContainerStarted","Data":"5b6ea630e32e3b2f4abc7538fc12b1972ae696fe5ac056765af0bd85b9b70268"} Nov 04 01:18:38.751536 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:38.751484 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podStartSLOduration=1.751468276 podStartE2EDuration="1.751468276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:38.749422927 +0000 UTC m=+1200.864533045" watchObservedRunningTime="2026-11-04 01:18:38.751468276 +0000 UTC m=+1200.866578396" Nov 04 01:18:38.930647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:38.930616 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:39.604597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:39.600283 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:39.610664 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:39.610620 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:40.411681 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:40.411638 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:40.889829 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:40.889796 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:40.902384 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:40.902355 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:41.118077 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.118046 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:41.475127 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.475092 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6"] Nov 04 01:18:41.478993 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.478974 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:41.483795 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.483775 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d7e45d\"/\"premium-w2-d7e45d-kserve-self-signed-certs\"" Nov 04 01:18:41.489191 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.489159 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6"] Nov 04 01:18:41.572007 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.571926 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:41.611083 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.611041 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-model-cache\") pod \"premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:41.611414 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.611391 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fv5n\" (UniqueName: \"kubernetes.io/projected/934d8fb0-d7fd-4091-9077-f8445780adf3-kube-api-access-4fv5n\") pod \"premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:41.612543 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.611617 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-tmp-dir\") pod \"premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:41.612543 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.611684 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/934d8fb0-d7fd-4091-9077-f8445780adf3-tls-certs\") pod \"premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:41.612543 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.611757 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-home\") pod \"premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:41.612543 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.611781 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-dshm\") pod \"premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:41.712453 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.712427 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fv5n\" (UniqueName: \"kubernetes.io/projected/934d8fb0-d7fd-4091-9077-f8445780adf3-kube-api-access-4fv5n\") pod \"premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:41.712633 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.712469 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-tmp-dir\") pod \"premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:41.712633 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.712489 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/934d8fb0-d7fd-4091-9077-f8445780adf3-tls-certs\") pod \"premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:41.712633 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.712529 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-home\") pod \"premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:41.712633 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.712544 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-dshm\") pod \"premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:41.712838 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.712723 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-model-cache\") pod \"premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:41.712888 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.712867 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-tmp-dir\") pod \"premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:41.712967 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.712945 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-home\") pod \"premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:41.713038 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.713019 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-model-cache\") pod \"premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:41.714959 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.714937 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-dshm\") pod \"premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:41.715168 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.715149 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/934d8fb0-d7fd-4091-9077-f8445780adf3-tls-certs\") pod \"premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:41.720920 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.720894 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fv5n\" (UniqueName: \"kubernetes.io/projected/934d8fb0-d7fd-4091-9077-f8445780adf3-kube-api-access-4fv5n\") pod \"premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:41.758015 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.757995 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:18:41.793493 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.793474 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:41.846742 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:41.846704 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:42.071901 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.071878 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6"] Nov 04 01:18:42.074501 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:18:42.074476 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod934d8fb0_d7fd_4091_9077_f8445780adf3.slice/crio-075647edf31c04b840f30986fd9c9e22f57881092b7d0faaead97d9b58cd0913 WatchSource:0}: Error finding container 075647edf31c04b840f30986fd9c9e22f57881092b7d0faaead97d9b58cd0913: Status 404 returned error can't find the container with id 075647edf31c04b840f30986fd9c9e22f57881092b7d0faaead97d9b58cd0913 Nov 04 01:18:42.454841 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.454754 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:42.693213 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.693176 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67"] Nov 04 01:18:42.698342 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.698317 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:42.701854 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.701829 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-099dd9\"/\"premium-w4-099dd9-kserve-self-signed-certs\"" Nov 04 01:18:42.710432 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.710411 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67"] Nov 04 01:18:42.760126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.760081 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" event={"ID":"934d8fb0-d7fd-4091-9077-f8445780adf3","Type":"ContainerStarted","Data":"50ba918b5aee6eeaeb7d9edaa4b2a0ab21e55c7e08d7a4ce9d0e7143e39d8cd6"} Nov 04 01:18:42.760377 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.760352 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" event={"ID":"934d8fb0-d7fd-4091-9077-f8445780adf3","Type":"ContainerStarted","Data":"075647edf31c04b840f30986fd9c9e22f57881092b7d0faaead97d9b58cd0913"} Nov 04 01:18:42.777011 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.776957 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podStartSLOduration=1.776941593 podStartE2EDuration="1.776941593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:42.775805052 +0000 UTC m=+1204.890915171" watchObservedRunningTime="2026-11-04 01:18:42.776941593 +0000 UTC m=+1204.892051713" Nov 04 01:18:42.793991 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.793967 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:42.820515 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.820484 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:42.824769 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.824746 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-tmp-dir\") pod \"premium-w4-099dd9-kserve-59cc96f9d5-q5c67\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:42.824873 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.824788 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-home\") pod \"premium-w4-099dd9-kserve-59cc96f9d5-q5c67\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:42.825046 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.825017 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv2cs\" (UniqueName: \"kubernetes.io/projected/9211f266-8d05-4bd8-9947-7893a9c0634b-kube-api-access-mv2cs\") pod \"premium-w4-099dd9-kserve-59cc96f9d5-q5c67\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:42.825109 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.825079 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9211f266-8d05-4bd8-9947-7893a9c0634b-tls-certs\") pod \"premium-w4-099dd9-kserve-59cc96f9d5-q5c67\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:42.825151 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.825136 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-model-cache\") pod \"premium-w4-099dd9-kserve-59cc96f9d5-q5c67\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:42.825192 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.825159 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-dshm\") pod \"premium-w4-099dd9-kserve-59cc96f9d5-q5c67\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:42.926029 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.925998 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv2cs\" (UniqueName: \"kubernetes.io/projected/9211f266-8d05-4bd8-9947-7893a9c0634b-kube-api-access-mv2cs\") pod \"premium-w4-099dd9-kserve-59cc96f9d5-q5c67\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:42.926029 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.926037 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9211f266-8d05-4bd8-9947-7893a9c0634b-tls-certs\") pod \"premium-w4-099dd9-kserve-59cc96f9d5-q5c67\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:42.926284 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.926076 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-model-cache\") pod \"premium-w4-099dd9-kserve-59cc96f9d5-q5c67\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:42.926284 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.926091 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-dshm\") pod \"premium-w4-099dd9-kserve-59cc96f9d5-q5c67\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:42.926284 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.926142 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-tmp-dir\") pod \"premium-w4-099dd9-kserve-59cc96f9d5-q5c67\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:42.926284 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.926183 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-home\") pod \"premium-w4-099dd9-kserve-59cc96f9d5-q5c67\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:42.926482 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.926451 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-model-cache\") pod \"premium-w4-099dd9-kserve-59cc96f9d5-q5c67\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:42.926540 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.926487 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-home\") pod \"premium-w4-099dd9-kserve-59cc96f9d5-q5c67\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:42.926626 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.926587 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-tmp-dir\") pod \"premium-w4-099dd9-kserve-59cc96f9d5-q5c67\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:42.928511 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.928484 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-dshm\") pod \"premium-w4-099dd9-kserve-59cc96f9d5-q5c67\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:42.929000 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.928981 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9211f266-8d05-4bd8-9947-7893a9c0634b-tls-certs\") pod \"premium-w4-099dd9-kserve-59cc96f9d5-q5c67\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:42.934030 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:42.934007 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv2cs\" (UniqueName: \"kubernetes.io/projected/9211f266-8d05-4bd8-9947-7893a9c0634b-kube-api-access-mv2cs\") pod \"premium-w4-099dd9-kserve-59cc96f9d5-q5c67\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:43.013890 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:43.013862 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:43.116892 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:43.116853 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:43.294906 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:43.294880 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67"] Nov 04 01:18:43.296075 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:18:43.296043 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9211f266_8d05_4bd8_9947_7893a9c0634b.slice/crio-3e20701e1773346ef3e8c2ec2bdd0ed39cfdf1ae138a1b059dc5ce08984d4b87 WatchSource:0}: Error finding container 3e20701e1773346ef3e8c2ec2bdd0ed39cfdf1ae138a1b059dc5ce08984d4b87: Status 404 returned error can't find the container with id 3e20701e1773346ef3e8c2ec2bdd0ed39cfdf1ae138a1b059dc5ce08984d4b87 Nov 04 01:18:43.765497 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:43.765464 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" event={"ID":"9211f266-8d05-4bd8-9947-7893a9c0634b","Type":"ContainerStarted","Data":"781b20931150a22ce2e7ce31ae0710f45917118500c93e1e9c319c5b3881a48f"} Nov 04 01:18:43.765895 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:43.765513 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" event={"ID":"9211f266-8d05-4bd8-9947-7893a9c0634b","Type":"ContainerStarted","Data":"3e20701e1773346ef3e8c2ec2bdd0ed39cfdf1ae138a1b059dc5ce08984d4b87"} Nov 04 01:18:43.782230 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:43.782162 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podStartSLOduration=1.782148981 podStartE2EDuration="1.782148981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:43.780863842 +0000 UTC m=+1205.895973959" watchObservedRunningTime="2026-11-04 01:18:43.782148981 +0000 UTC m=+1205.897259098" Nov 04 01:18:43.826672 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:43.826648 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:44.014611 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:44.014577 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:44.594823 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:44.594787 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:45.329597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:45.328677 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:46.029223 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:46.029184 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:46.937291 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:46.937264 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:47.599968 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:47.599936 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:47.618118 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:47.618093 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:47.732444 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:47.731813 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:47.802413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:47.802387 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:18:48.521664 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:48.521628 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:49.528497 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:49.528419 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:50.324804 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:50.324748 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:51.026776 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.026739 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:51.068837 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.068805 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6"] Nov 04 01:18:51.072991 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.072963 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:18:51.077961 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.077937 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-9c8bf5\"/\"kube-root-ca.crt\"" Nov 04 01:18:51.079228 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.079204 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-9c8bf5\"/\"default-dockercfg-q6h84\"" Nov 04 01:18:51.079352 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.079228 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-9c8bf5\"/\"premium-w5-9c8bf5-kserve-self-signed-certs\"" Nov 04 01:18:51.082047 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.082029 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-9c8bf5\"/\"openshift-service-ca.crt\"" Nov 04 01:18:51.083774 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.083753 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6"] Nov 04 01:18:51.193635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.193599 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:51.218588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.216153 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-home\") pod \"premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:18:51.218588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.216238 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-tmp-dir\") pod \"premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:18:51.218588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.216301 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnddd\" (UniqueName: \"kubernetes.io/projected/1ce1f2a5-3954-4044-a85e-2761a6b2088e-kube-api-access-lnddd\") pod \"premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:18:51.218588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.216330 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-model-cache\") pod \"premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:18:51.218588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.216409 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-dshm\") pod \"premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:18:51.218588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.216440 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ce1f2a5-3954-4044-a85e-2761a6b2088e-tls-certs\") pod \"premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:18:51.317372 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.317294 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-dshm\") pod \"premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:18:51.317372 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.317331 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ce1f2a5-3954-4044-a85e-2761a6b2088e-tls-certs\") pod \"premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:18:51.317603 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.317381 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-home\") pod \"premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:18:51.317603 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.317411 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-tmp-dir\") pod \"premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:18:51.317603 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.317448 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnddd\" (UniqueName: \"kubernetes.io/projected/1ce1f2a5-3954-4044-a85e-2761a6b2088e-kube-api-access-lnddd\") pod \"premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:18:51.317603 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.317478 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-model-cache\") pod \"premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:18:51.317810 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.317800 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-tmp-dir\") pod \"premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:18:51.317893 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.317873 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-home\") pod \"premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:18:51.317949 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.317889 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-model-cache\") pod \"premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:18:51.319866 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.319838 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-dshm\") pod \"premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:18:51.320090 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.320073 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ce1f2a5-3954-4044-a85e-2761a6b2088e-tls-certs\") pod \"premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:18:51.326328 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.326308 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnddd\" (UniqueName: \"kubernetes.io/projected/1ce1f2a5-3954-4044-a85e-2761a6b2088e-kube-api-access-lnddd\") pod \"premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:18:51.387656 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.387634 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:18:51.669886 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.669868 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6"] Nov 04 01:18:51.672665 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:18:51.672629 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ce1f2a5_3954_4044_a85e_2761a6b2088e.slice/crio-79d0fed5bb2fe15594bb27cfbb914c57789b21aea54182302740e78991df0e21 WatchSource:0}: Error finding container 79d0fed5bb2fe15594bb27cfbb914c57789b21aea54182302740e78991df0e21: Status 404 returned error can't find the container with id 79d0fed5bb2fe15594bb27cfbb914c57789b21aea54182302740e78991df0e21 Nov 04 01:18:51.721298 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.721275 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:51.794145 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.794105 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:51.806987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.806952 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" event={"ID":"1ce1f2a5-3954-4044-a85e-2761a6b2088e","Type":"ContainerStarted","Data":"8cfc22a3a39a4a06c6498abbbec07f492d4961501df585f5f816bfe7674349dc"} Nov 04 01:18:51.806987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.806994 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" event={"ID":"1ce1f2a5-3954-4044-a85e-2761a6b2088e","Type":"ContainerStarted","Data":"79d0fed5bb2fe15594bb27cfbb914c57789b21aea54182302740e78991df0e21"} Nov 04 01:18:51.808830 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.808809 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:51.820253 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.820237 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:18:51.827663 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:51.827624 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podStartSLOduration=0.827610707 podStartE2EDuration="827.610707ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:51.826298622 +0000 UTC m=+1213.941408739" watchObservedRunningTime="2026-11-04 01:18:51.827610707 +0000 UTC m=+1213.942720826" Nov 04 01:18:52.388174 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:52.388141 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:18:52.524170 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:52.524139 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:53.015022 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:53.014987 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:53.028002 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:53.027968 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:53.249125 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:53.249088 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:53.343663 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:53.343588 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:18:53.826822 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:53.826796 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:18:54.720241 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:54.720210 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:55.245818 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:55.245778 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:55.945212 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:55.945182 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:56.005283 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.005241 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx"] Nov 04 01:18:56.011479 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.011453 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:18:56.019200 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.019175 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx"] Nov 04 01:18:56.024166 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.024145 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ca55c1\"/\"premium-w3-ca55c1-kserve-self-signed-certs\"" Nov 04 01:18:56.160587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.157531 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:56.171159 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.171121 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-model-cache\") pod \"premium-w3-ca55c1-kserve-79dd86bcc-wx5qx\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:18:56.171381 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.171197 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-tmp-dir\") pod \"premium-w3-ca55c1-kserve-79dd86bcc-wx5qx\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:18:56.171381 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.171290 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g9w6\" (UniqueName: \"kubernetes.io/projected/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-kube-api-access-4g9w6\") pod \"premium-w3-ca55c1-kserve-79dd86bcc-wx5qx\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:18:56.171381 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.171317 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-dshm\") pod \"premium-w3-ca55c1-kserve-79dd86bcc-wx5qx\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:18:56.171381 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.171352 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-tls-certs\") pod \"premium-w3-ca55c1-kserve-79dd86bcc-wx5qx\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:18:56.171381 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.171374 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-home\") pod \"premium-w3-ca55c1-kserve-79dd86bcc-wx5qx\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:18:56.272438 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.272401 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g9w6\" (UniqueName: \"kubernetes.io/projected/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-kube-api-access-4g9w6\") pod \"premium-w3-ca55c1-kserve-79dd86bcc-wx5qx\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:18:56.272438 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.272442 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-dshm\") pod \"premium-w3-ca55c1-kserve-79dd86bcc-wx5qx\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:18:56.273012 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.272478 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-tls-certs\") pod \"premium-w3-ca55c1-kserve-79dd86bcc-wx5qx\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:18:56.273012 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.272500 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-home\") pod \"premium-w3-ca55c1-kserve-79dd86bcc-wx5qx\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:18:56.273012 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.272577 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-model-cache\") pod \"premium-w3-ca55c1-kserve-79dd86bcc-wx5qx\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:18:56.273012 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.272644 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-tmp-dir\") pod \"premium-w3-ca55c1-kserve-79dd86bcc-wx5qx\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:18:56.273012 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.272962 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-home\") pod \"premium-w3-ca55c1-kserve-79dd86bcc-wx5qx\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:18:56.273547 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.273035 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-model-cache\") pod \"premium-w3-ca55c1-kserve-79dd86bcc-wx5qx\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:18:56.273547 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.273082 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-tmp-dir\") pod \"premium-w3-ca55c1-kserve-79dd86bcc-wx5qx\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:18:56.275108 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.275082 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-dshm\") pod \"premium-w3-ca55c1-kserve-79dd86bcc-wx5qx\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:18:56.275218 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.275200 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-tls-certs\") pod \"premium-w3-ca55c1-kserve-79dd86bcc-wx5qx\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:18:56.285883 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.285862 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g9w6\" (UniqueName: \"kubernetes.io/projected/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-kube-api-access-4g9w6\") pod \"premium-w3-ca55c1-kserve-79dd86bcc-wx5qx\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:18:56.329766 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.329735 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:18:56.613148 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.613120 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx"] Nov 04 01:18:56.615482 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:18:56.615455 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7025b15d_c99e_4ccb_98a2_2a6f4cc69a1d.slice/crio-6dd782dd2d2f79ab1c5b9e8e08cf1c746536932194fc7582d5a1a6c6fc137346 WatchSource:0}: Error finding container 6dd782dd2d2f79ab1c5b9e8e08cf1c746536932194fc7582d5a1a6c6fc137346: Status 404 returned error can't find the container with id 6dd782dd2d2f79ab1c5b9e8e08cf1c746536932194fc7582d5a1a6c6fc137346 Nov 04 01:18:56.830697 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.830114 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:56.840611 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.840546 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" event={"ID":"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d","Type":"ContainerStarted","Data":"c09853f05e48f4f115111700306daa592e7d322ddb4eebf9077cdb95b26c8adc"} Nov 04 01:18:56.840785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.840628 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" event={"ID":"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d","Type":"ContainerStarted","Data":"6dd782dd2d2f79ab1c5b9e8e08cf1c746536932194fc7582d5a1a6c6fc137346"} Nov 04 01:18:56.861958 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:56.861909 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podStartSLOduration=1.861895707 podStartE2EDuration="1.861895707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:56.860468196 +0000 UTC m=+1218.975578304" watchObservedRunningTime="2026-11-04 01:18:56.861895707 +0000 UTC m=+1218.977005824" Nov 04 01:18:57.330558 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:57.330520 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:18:57.617945 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:57.617846 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:58.317258 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:58.316929 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:58.955294 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:58.955259 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845"] Nov 04 01:18:58.958935 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:58.958916 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:18:58.968048 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:58.968009 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-499ddd\"/\"kube-root-ca.crt\"" Nov 04 01:18:58.968276 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:58.968138 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-499ddd\"/\"default-dockercfg-vczgh\"" Nov 04 01:18:58.969237 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:58.969219 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-499ddd\"/\"openshift-service-ca.crt\"" Nov 04 01:18:58.969429 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:58.969374 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-499ddd\"/\"premium-w6-499ddd-kserve-self-signed-certs\"" Nov 04 01:18:58.972115 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:58.972089 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845"] Nov 04 01:18:58.999827 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:58.999783 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-tmp-dir\") pod \"premium-w6-499ddd-kserve-76554bcc7d-nz845\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:18:58.999966 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:58.999840 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-home\") pod \"premium-w6-499ddd-kserve-76554bcc7d-nz845\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:18:59.000006 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:58.999961 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/790de7b2-be85-47bb-b2f2-d21b1422bac9-tls-certs\") pod \"premium-w6-499ddd-kserve-76554bcc7d-nz845\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:18:59.000006 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:58.999993 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-model-cache\") pod \"premium-w6-499ddd-kserve-76554bcc7d-nz845\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:18:59.000077 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:59.000016 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsz4m\" (UniqueName: \"kubernetes.io/projected/790de7b2-be85-47bb-b2f2-d21b1422bac9-kube-api-access-hsz4m\") pod \"premium-w6-499ddd-kserve-76554bcc7d-nz845\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:18:59.000110 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:59.000094 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-dshm\") pod \"premium-w6-499ddd-kserve-76554bcc7d-nz845\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:18:59.049129 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:59.049090 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:59.100818 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:59.100787 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/790de7b2-be85-47bb-b2f2-d21b1422bac9-tls-certs\") pod \"premium-w6-499ddd-kserve-76554bcc7d-nz845\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:18:59.100959 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:59.100841 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-model-cache\") pod \"premium-w6-499ddd-kserve-76554bcc7d-nz845\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:18:59.100959 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:59.100869 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsz4m\" (UniqueName: \"kubernetes.io/projected/790de7b2-be85-47bb-b2f2-d21b1422bac9-kube-api-access-hsz4m\") pod \"premium-w6-499ddd-kserve-76554bcc7d-nz845\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:18:59.100959 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:59.100918 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-dshm\") pod \"premium-w6-499ddd-kserve-76554bcc7d-nz845\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:18:59.101104 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:59.100989 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-tmp-dir\") pod \"premium-w6-499ddd-kserve-76554bcc7d-nz845\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:18:59.101104 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:59.101013 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-home\") pod \"premium-w6-499ddd-kserve-76554bcc7d-nz845\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:18:59.101408 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:59.101380 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-model-cache\") pod \"premium-w6-499ddd-kserve-76554bcc7d-nz845\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:18:59.101740 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:59.101418 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-tmp-dir\") pod \"premium-w6-499ddd-kserve-76554bcc7d-nz845\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:18:59.101740 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:59.101442 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-home\") pod \"premium-w6-499ddd-kserve-76554bcc7d-nz845\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:18:59.103501 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:59.103478 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-dshm\") pod \"premium-w6-499ddd-kserve-76554bcc7d-nz845\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:18:59.103640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:59.103624 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/790de7b2-be85-47bb-b2f2-d21b1422bac9-tls-certs\") pod \"premium-w6-499ddd-kserve-76554bcc7d-nz845\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:18:59.108776 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:59.108751 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsz4m\" (UniqueName: \"kubernetes.io/projected/790de7b2-be85-47bb-b2f2-d21b1422bac9-kube-api-access-hsz4m\") pod \"premium-w6-499ddd-kserve-76554bcc7d-nz845\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:18:59.188964 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:59.188918 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:59.275686 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:59.275656 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:18:59.549778 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:59.549733 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845"] Nov 04 01:18:59.552285 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:18:59.552258 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod790de7b2_be85_47bb_b2f2_d21b1422bac9.slice/crio-eebe9fe0df273524f1e51646195558b6aa46e75b7280de2fbd04d577a19debc2 WatchSource:0}: Error finding container eebe9fe0df273524f1e51646195558b6aa46e75b7280de2fbd04d577a19debc2: Status 404 returned error can't find the container with id eebe9fe0df273524f1e51646195558b6aa46e75b7280de2fbd04d577a19debc2 Nov 04 01:18:59.843638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:59.842838 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:18:59.869962 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:59.869931 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" event={"ID":"790de7b2-be85-47bb-b2f2-d21b1422bac9","Type":"ContainerStarted","Data":"e3260569ca48be59369e194750aa3a7ffc4a75f1c966ad60618289acd6c9e8f1"} Nov 04 01:18:59.869962 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:59.869970 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" event={"ID":"790de7b2-be85-47bb-b2f2-d21b1422bac9","Type":"ContainerStarted","Data":"eebe9fe0df273524f1e51646195558b6aa46e75b7280de2fbd04d577a19debc2"} Nov 04 01:18:59.884547 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:18:59.884515 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:18:59.888534 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:18:59.888492 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podStartSLOduration=1.888478812 podStartE2EDuration="1.888478812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:59.887928518 +0000 UTC m=+1222.003038648" watchObservedRunningTime="2026-11-04 01:18:59.888478812 +0000 UTC m=+1222.003588931" Nov 04 01:19:00.276809 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:00.276777 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:19:00.808969 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:00.808916 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:01.388636 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:01.388606 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:19:01.401378 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:01.401348 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:19:01.534541 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:01.526327 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:01.892444 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:01.892414 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:19:02.330588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:02.326873 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:02.944234 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:02.944077 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:03.629587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:03.625925 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:04.314595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:04.312590 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:04.640420 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:04.640340 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:19:04.866942 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:04.866910 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:05.060721 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:05.060678 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:06.002185 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.002151 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq"] Nov 04 01:19:06.006451 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.006432 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:06.014079 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.013288 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-586541\"/\"default-dockercfg-26p4v\"" Nov 04 01:19:06.015418 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.015393 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-586541\"/\"premium-w1-586541-kserve-self-signed-certs\"" Nov 04 01:19:06.015687 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.015667 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq"] Nov 04 01:19:06.015929 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.015911 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-586541\"/\"openshift-service-ca.crt\"" Nov 04 01:19:06.016098 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.016025 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-586541\"/\"kube-root-ca.crt\"" Nov 04 01:19:06.065766 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.065729 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-model-cache\") pod \"premium-w1-586541-kserve-6df644b944-987zq\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:06.065766 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.065769 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-dshm\") pod \"premium-w1-586541-kserve-6df644b944-987zq\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:06.065976 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.065954 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-home\") pod \"premium-w1-586541-kserve-6df644b944-987zq\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:06.066164 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.066148 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-tmp-dir\") pod \"premium-w1-586541-kserve-6df644b944-987zq\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:06.066234 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.066183 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761be4d8-0736-4a22-bd18-2729dbc85a11-tls-certs\") pod \"premium-w1-586541-kserve-6df644b944-987zq\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:06.066234 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.066225 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p66jd\" (UniqueName: \"kubernetes.io/projected/761be4d8-0736-4a22-bd18-2729dbc85a11-kube-api-access-p66jd\") pod \"premium-w1-586541-kserve-6df644b944-987zq\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:06.083784 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.083753 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:06.168016 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.167237 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-home\") pod \"premium-w1-586541-kserve-6df644b944-987zq\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:06.168016 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.167300 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-tmp-dir\") pod \"premium-w1-586541-kserve-6df644b944-987zq\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:06.168016 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.167348 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761be4d8-0736-4a22-bd18-2729dbc85a11-tls-certs\") pod \"premium-w1-586541-kserve-6df644b944-987zq\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:06.168016 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.167387 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p66jd\" (UniqueName: \"kubernetes.io/projected/761be4d8-0736-4a22-bd18-2729dbc85a11-kube-api-access-p66jd\") pod \"premium-w1-586541-kserve-6df644b944-987zq\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:06.168016 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.167480 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-model-cache\") pod \"premium-w1-586541-kserve-6df644b944-987zq\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:06.168016 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.167512 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-dshm\") pod \"premium-w1-586541-kserve-6df644b944-987zq\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:06.170635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.169158 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-home\") pod \"premium-w1-586541-kserve-6df644b944-987zq\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:06.170635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.169402 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-model-cache\") pod \"premium-w1-586541-kserve-6df644b944-987zq\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:06.170635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.169660 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-tmp-dir\") pod \"premium-w1-586541-kserve-6df644b944-987zq\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:06.172322 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.172299 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761be4d8-0736-4a22-bd18-2729dbc85a11-tls-certs\") pod \"premium-w1-586541-kserve-6df644b944-987zq\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:06.174193 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.174168 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-dshm\") pod \"premium-w1-586541-kserve-6df644b944-987zq\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:06.190027 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.190001 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p66jd\" (UniqueName: \"kubernetes.io/projected/761be4d8-0736-4a22-bd18-2729dbc85a11-kube-api-access-p66jd\") pod \"premium-w1-586541-kserve-6df644b944-987zq\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:06.321990 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.321960 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:06.330272 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.330248 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:19:06.343668 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.343648 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:19:06.407681 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.407617 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:06.601016 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.600993 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq"] Nov 04 01:19:06.602815 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:19:06.602789 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod761be4d8_0736_4a22_bd18_2729dbc85a11.slice/crio-254c73d15ae1233748a9a67ac7fa371a804bb3bf994e42f616a602d2ab4f4bbf WatchSource:0}: Error finding container 254c73d15ae1233748a9a67ac7fa371a804bb3bf994e42f616a602d2ab4f4bbf: Status 404 returned error can't find the container with id 254c73d15ae1233748a9a67ac7fa371a804bb3bf994e42f616a602d2ab4f4bbf Nov 04 01:19:06.925677 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.925590 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" event={"ID":"761be4d8-0736-4a22-bd18-2729dbc85a11","Type":"ContainerStarted","Data":"6e4774dbed44e28893d7cc9d66e85863f7b972b92ba739a6336a66d7454d9809"} Nov 04 01:19:06.925677 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.925642 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" event={"ID":"761be4d8-0736-4a22-bd18-2729dbc85a11","Type":"ContainerStarted","Data":"254c73d15ae1233748a9a67ac7fa371a804bb3bf994e42f616a602d2ab4f4bbf"} Nov 04 01:19:06.937288 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.937265 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:19:06.945180 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:06.945139 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podStartSLOduration=1.945127044 podStartE2EDuration="1.945127044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:06.943717555 +0000 UTC m=+1229.058827675" watchObservedRunningTime="2026-11-04 01:19:06.945127044 +0000 UTC m=+1229.060237161" Nov 04 01:19:07.109544 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:07.109490 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:07.322805 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:07.322756 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:08.126612 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:08.126369 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:08.831211 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:08.831165 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:09.276328 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:09.276304 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:19:09.288807 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:09.288785 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:19:09.527736 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:09.527644 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:09.952839 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:09.952771 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:19:10.227482 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:10.227386 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:11.120340 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:11.120306 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:11.830691 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:11.830648 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:12.513342 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.506802 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4"] Nov 04 01:19:12.520995 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.520968 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.538607 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.537498 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-d7e45d-openshift-default-dockercfg-rl5m5\"" Nov 04 01:19:12.538858 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.538710 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:12.541796 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.541773 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4"] Nov 04 01:19:12.631054 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.631013 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a7dd85-4ad8-4990-8907-cb64e413cceb-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.631054 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.631057 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.631235 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.631163 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ntjp\" (UniqueName: \"kubernetes.io/projected/82a7dd85-4ad8-4990-8907-cb64e413cceb-kube-api-access-5ntjp\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.631235 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.631191 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.631315 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.631242 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.631315 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.631259 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.631385 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.631339 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.631423 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.631376 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.631461 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.631440 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.732153 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.732123 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.732153 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.732154 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.732356 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.732191 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.732356 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.732250 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a7dd85-4ad8-4990-8907-cb64e413cceb-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.732356 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.732276 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.732356 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.732329 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ntjp\" (UniqueName: \"kubernetes.io/projected/82a7dd85-4ad8-4990-8907-cb64e413cceb-kube-api-access-5ntjp\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.732557 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.732361 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.732557 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.732407 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.732557 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.732437 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.732557 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.732551 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.732826 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.732736 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.732900 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.732877 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.733153 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.733132 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a7dd85-4ad8-4990-8907-cb64e413cceb-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.733211 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.733167 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.734759 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.734734 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.734901 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.734884 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.740203 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.740177 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.740298 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.740266 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ntjp\" (UniqueName: \"kubernetes.io/projected/82a7dd85-4ad8-4990-8907-cb64e413cceb-kube-api-access-5ntjp\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:12.853356 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:12.853296 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:13.130033 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:13.130009 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4"] Nov 04 01:19:13.131970 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:19:13.131940 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a7dd85_4ad8_4990_8907_cb64e413cceb.slice/crio-1fd54b535ae3cecb6dd3e5c7c86fcdeca7e8748d7f4e1eddf6e31729889889b7 WatchSource:0}: Error finding container 1fd54b535ae3cecb6dd3e5c7c86fcdeca7e8748d7f4e1eddf6e31729889889b7: Status 404 returned error can't find the container with id 1fd54b535ae3cecb6dd3e5c7c86fcdeca7e8748d7f4e1eddf6e31729889889b7 Nov 04 01:19:13.134909 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:13.134876 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:13.134994 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:13.134940 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:13.134994 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:13.134971 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:13.317459 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:13.317202 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:13.992807 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:13.992768 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" event={"ID":"82a7dd85-4ad8-4990-8907-cb64e413cceb","Type":"ContainerStarted","Data":"bd9cbabd2ff2ea49ee861aea08045312dc508f5760614c80ead0293e4dcc7915"} Nov 04 01:19:13.993195 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:13.992828 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" event={"ID":"82a7dd85-4ad8-4990-8907-cb64e413cceb","Type":"ContainerStarted","Data":"1fd54b535ae3cecb6dd3e5c7c86fcdeca7e8748d7f4e1eddf6e31729889889b7"} Nov 04 01:19:14.011519 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:14.011471 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" podStartSLOduration=2.011459352 podStartE2EDuration="2.011459352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:14.01003504 +0000 UTC m=+1236.125145157" watchObservedRunningTime="2026-11-04 01:19:14.011459352 +0000 UTC m=+1236.126569470" Nov 04 01:19:14.033922 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:14.033891 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:14.731429 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:14.731368 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:14.854027 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:14.853999 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:14.855430 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:14.855408 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 04 01:19:14.855539 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:14.855453 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" podUID="82a7dd85-4ad8-4990-8907-cb64e413cceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 04 01:19:15.446253 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:15.446214 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:15.854538 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:15.854497 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 04 01:19:15.854715 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:15.854585 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" podUID="82a7dd85-4ad8-4990-8907-cb64e413cceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 04 01:19:16.323084 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:16.323048 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:16.336878 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:16.336853 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:16.444720 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:16.444672 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:16.854389 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:16.854357 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 04 01:19:16.854814 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:16.854412 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" podUID="82a7dd85-4ad8-4990-8907-cb64e413cceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 04 01:19:17.014143 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:17.014109 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:17.140362 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:17.140293 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:17.853948 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:17.853916 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 04 01:19:17.854128 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:17.853971 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" podUID="82a7dd85-4ad8-4990-8907-cb64e413cceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 04 01:19:18.029854 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:18.029820 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:18.855030 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:18.854997 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 04 01:19:18.855296 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:18.855271 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" podUID="82a7dd85-4ad8-4990-8907-cb64e413cceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 04 01:19:19.148373 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:19.148267 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:19.880759 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:19.880728 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:19.881074 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:19.881038 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:19.882016 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:19.881993 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:19:19.958376 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:19.958339 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:20.866948 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:20.866915 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:21.862843 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:21.862807 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:23.045505 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:23.043995 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:24.043715 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:24.043676 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:24.754751 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:24.754719 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:25.474845 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:25.474811 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:26.389598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:26.389525 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:26.950284 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:26.950240 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:27.761181 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:27.761147 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:28.537064 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:28.537017 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:29.334442 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:29.334373 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:29.866557 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:19:29.866513 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:19:30.037247 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:30.037213 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:30.607896 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:30.607866 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:30.759365 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:30.759292 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:31.426508 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:31.426473 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:32.119450 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:32.119403 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:32.840143 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:32.840111 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:33.027245 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:33.027211 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:19:33.560899 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:33.560868 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:34.321777 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:34.321172 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:35.039130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:35.039077 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:35.841609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:35.840918 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:36.737644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:36.737609 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:37.506077 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:37.506040 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:38.223195 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:38.223145 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:38.928006 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:38.927902 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:39.707165 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:39.707128 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:40.371330 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:40.371295 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:41.018318 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:41.018261 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:41.716673 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:41.716633 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:41.922758 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:41.922725 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l"] Nov 04 01:19:41.928668 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:41.928644 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:41.940087 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:41.940056 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l"] Nov 04 01:19:41.986119 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:41.986055 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:41.986119 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:41.986086 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:41.986277 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:41.986163 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:41.986277 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:41.986195 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:41.986277 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:41.986227 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs2l6\" (UniqueName: \"kubernetes.io/projected/d26cecde-b16d-4fe1-add0-41a6d0a5301a-kube-api-access-xs2l6\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:41.986376 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:41.986347 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:41.986376 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:41.986372 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:41.986437 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:41.986409 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:41.986487 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:41.986474 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:42.019492 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.019469 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-9c8bf5-openshift-default-dockercfg-4ks7x\"" Nov 04 01:19:42.087455 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.087427 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:42.087637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.087474 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:42.087637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.087506 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:42.087637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.087525 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:42.087637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.087554 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:42.087637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.087621 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:42.087912 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.087745 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs2l6\" (UniqueName: \"kubernetes.io/projected/d26cecde-b16d-4fe1-add0-41a6d0a5301a-kube-api-access-xs2l6\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:42.087912 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.087838 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:42.087912 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.087869 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:42.088096 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.087971 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:42.088152 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.088117 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:42.088283 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.088262 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:42.088421 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.088401 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:42.088557 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.088535 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:42.089940 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.089917 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:42.090134 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.090114 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:42.095842 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.095816 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:42.095929 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.095859 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs2l6\" (UniqueName: \"kubernetes.io/projected/d26cecde-b16d-4fe1-add0-41a6d0a5301a-kube-api-access-xs2l6\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:42.249587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.249497 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:42.261716 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.261694 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l"] Nov 04 01:19:42.265576 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.265539 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.270555 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.270527 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-499ddd-openshift-default-dockercfg-fb8nq\"" Nov 04 01:19:42.278118 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.278097 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l"] Nov 04 01:19:42.289729 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.289700 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.289857 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.289748 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.289857 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.289802 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.289857 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.289844 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.290027 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.289877 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.290027 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.289912 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36db7091-781e-4f96-a979-6ad25e0fb53f-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.290027 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.289981 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-474bf\" (UniqueName: \"kubernetes.io/projected/36db7091-781e-4f96-a979-6ad25e0fb53f-kube-api-access-474bf\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.290175 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.290041 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.290175 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.290103 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.385394 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.385359 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf"] Nov 04 01:19:42.389897 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.389875 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.390645 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.390611 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.390768 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.390651 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.390768 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.390680 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.390768 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.390698 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36db7091-781e-4f96-a979-6ad25e0fb53f-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.390768 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.390747 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-474bf\" (UniqueName: \"kubernetes.io/projected/36db7091-781e-4f96-a979-6ad25e0fb53f-kube-api-access-474bf\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.391022 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.390798 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.391022 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.390845 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.391022 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.390887 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.391022 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.390918 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.391236 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.391161 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.391236 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.391224 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.391417 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.391392 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.391524 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.391418 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.391677 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.391661 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36db7091-781e-4f96-a979-6ad25e0fb53f-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.393421 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.393402 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.393691 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.393675 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.401270 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.401226 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf"] Nov 04 01:19:42.410936 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.410911 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.411250 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.411230 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-474bf\" (UniqueName: \"kubernetes.io/projected/36db7091-781e-4f96-a979-6ad25e0fb53f-kube-api-access-474bf\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.470358 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.470329 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw"] Nov 04 01:19:42.470829 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.470801 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ca55c1-openshift-default-dockercfg-t5hrc\"" Nov 04 01:19:42.474093 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.474075 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.484781 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.484752 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw"] Nov 04 01:19:42.488638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.488604 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:42.491921 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.491896 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.492018 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.491952 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzr2z\" (UniqueName: \"kubernetes.io/projected/169d147a-6807-4836-8eca-4b7aa3d33bd5-kube-api-access-lzr2z\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.492018 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.491975 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.492018 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.492013 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.492138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.492040 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.492138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.492060 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.492138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.492079 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.492138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.492109 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.492138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.492129 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.492332 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.492149 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.492332 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.492173 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.492332 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.492194 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.492332 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.492223 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.492332 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.492247 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.492332 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.492288 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169d147a-6807-4836-8eca-4b7aa3d33bd5-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.492332 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.492307 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8wmd\" (UniqueName: \"kubernetes.io/projected/8c157ec5-7fa3-4774-bace-71cd27ee9d82-kube-api-access-f8wmd\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.492332 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.492325 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.492770 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.492347 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.539859 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.539834 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l"] Nov 04 01:19:42.541301 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:19:42.541279 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26cecde_b16d_4fe1_add0_41a6d0a5301a.slice/crio-fbfa0fa11cc19a966e687b7e5a97f6de9a180a3e08d5da107328232bfe3810d6 WatchSource:0}: Error finding container fbfa0fa11cc19a966e687b7e5a97f6de9a180a3e08d5da107328232bfe3810d6: Status 404 returned error can't find the container with id fbfa0fa11cc19a966e687b7e5a97f6de9a180a3e08d5da107328232bfe3810d6 Nov 04 01:19:42.543653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.543624 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:42.543708 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.543686 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:42.543752 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.543715 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:42.571748 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.571729 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-586541-openshift-default-dockercfg-tgz2p\"" Nov 04 01:19:42.592514 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.592493 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:42.592876 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.592852 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169d147a-6807-4836-8eca-4b7aa3d33bd5-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.592935 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.592897 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8wmd\" (UniqueName: \"kubernetes.io/projected/8c157ec5-7fa3-4774-bace-71cd27ee9d82-kube-api-access-f8wmd\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.592970 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.592937 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.593010 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.592976 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.593066 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.593008 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.593120 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.593064 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzr2z\" (UniqueName: \"kubernetes.io/projected/169d147a-6807-4836-8eca-4b7aa3d33bd5-kube-api-access-lzr2z\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.593120 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.593095 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.593223 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.593133 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.593223 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.593188 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.593327 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.593220 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.593327 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.593247 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.593327 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.593302 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.593505 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.593333 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.593505 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.593356 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.593505 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.593393 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.593505 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.593418 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.593505 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.593426 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.593505 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.593439 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.593505 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.593504 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.594099 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.593544 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.594099 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.593619 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.594099 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.593670 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169d147a-6807-4836-8eca-4b7aa3d33bd5-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.594099 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.593731 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.594313 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.594159 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.594462 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.594444 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.594768 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.594744 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.595036 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.595014 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.595258 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.595238 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.596194 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.596163 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.596626 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.596610 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.596756 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.596737 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.597754 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.597736 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.602470 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.602444 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8wmd\" (UniqueName: \"kubernetes.io/projected/8c157ec5-7fa3-4774-bace-71cd27ee9d82-kube-api-access-f8wmd\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.602643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.602619 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.603368 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.603352 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzr2z\" (UniqueName: \"kubernetes.io/projected/169d147a-6807-4836-8eca-4b7aa3d33bd5-kube-api-access-lzr2z\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.604583 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.604544 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.670867 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.670773 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:42.718524 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.718493 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:42.797121 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.796863 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:42.896113 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.895407 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l"] Nov 04 01:19:42.897666 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:19:42.897629 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36db7091_781e_4f96_a979_6ad25e0fb53f.slice/crio-b57b55063da83f979951968c4d7e49da118cd6cb13de890da8c35eb837bcd257 WatchSource:0}: Error finding container b57b55063da83f979951968c4d7e49da118cd6cb13de890da8c35eb837bcd257: Status 404 returned error can't find the container with id b57b55063da83f979951968c4d7e49da118cd6cb13de890da8c35eb837bcd257 Nov 04 01:19:42.901649 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.901609 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:42.901719 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.901704 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:42.901764 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:42.901748 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:43.010754 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.010731 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf"] Nov 04 01:19:43.014639 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:19:43.013875 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c157ec5_7fa3_4774_bace_71cd27ee9d82.slice/crio-ebc7b1771464d3567a87837fe18c475e0c6359b9398dd6f632d21412af77eb6b WatchSource:0}: Error finding container ebc7b1771464d3567a87837fe18c475e0c6359b9398dd6f632d21412af77eb6b: Status 404 returned error can't find the container with id ebc7b1771464d3567a87837fe18c475e0c6359b9398dd6f632d21412af77eb6b Nov 04 01:19:43.017549 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.017505 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:43.017663 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.017634 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:43.017720 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.017682 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:43.102230 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.101623 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw"] Nov 04 01:19:43.107726 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.107191 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:43.107726 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.107280 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:43.107726 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.107316 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:43.177806 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.176796 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" event={"ID":"d26cecde-b16d-4fe1-add0-41a6d0a5301a","Type":"ContainerStarted","Data":"08d2bc451cb6986a1954ebf9622dbb315cb8fccc8772f554e26c0f33ee500a44"} Nov 04 01:19:43.177806 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.176862 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" event={"ID":"d26cecde-b16d-4fe1-add0-41a6d0a5301a","Type":"ContainerStarted","Data":"fbfa0fa11cc19a966e687b7e5a97f6de9a180a3e08d5da107328232bfe3810d6"} Nov 04 01:19:43.194610 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.184462 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" event={"ID":"8c157ec5-7fa3-4774-bace-71cd27ee9d82","Type":"ContainerStarted","Data":"ebc7b1771464d3567a87837fe18c475e0c6359b9398dd6f632d21412af77eb6b"} Nov 04 01:19:43.194610 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.186082 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" event={"ID":"169d147a-6807-4836-8eca-4b7aa3d33bd5","Type":"ContainerStarted","Data":"c83179357a361ece76faea6859d63c42fd4b958948cacd01825356758f719e29"} Nov 04 01:19:43.194610 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.193703 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" event={"ID":"36db7091-781e-4f96-a979-6ad25e0fb53f","Type":"ContainerStarted","Data":"051f67ebf8605d67633e79cdcc778071025ef0d5567155753fdec4b2613434b0"} Nov 04 01:19:43.194610 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.193765 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" event={"ID":"36db7091-781e-4f96-a979-6ad25e0fb53f","Type":"ContainerStarted","Data":"b57b55063da83f979951968c4d7e49da118cd6cb13de890da8c35eb837bcd257"} Nov 04 01:19:43.206841 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.206690 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" podStartSLOduration=2.206675424 podStartE2EDuration="2.206675424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:43.201254942 +0000 UTC m=+1265.316365060" watchObservedRunningTime="2026-11-04 01:19:43.206675424 +0000 UTC m=+1265.321785541" Nov 04 01:19:43.231966 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.231907 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" podStartSLOduration=1.231888191 podStartE2EDuration="1.231888191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:43.22623818 +0000 UTC m=+1265.341348300" watchObservedRunningTime="2026-11-04 01:19:43.231888191 +0000 UTC m=+1265.346998311" Nov 04 01:19:43.250143 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.250118 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:43.251318 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.251075 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 04 01:19:43.251318 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.251116 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" podUID="d26cecde-b16d-4fe1-add0-41a6d0a5301a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 04 01:19:43.595023 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.594944 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:43.597346 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.597018 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 01:19:43.597346 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.597073 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" podUID="36db7091-781e-4f96-a979-6ad25e0fb53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 01:19:43.636590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:43.633044 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:44.235592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:44.229096 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" event={"ID":"8c157ec5-7fa3-4774-bace-71cd27ee9d82","Type":"ContainerStarted","Data":"cc4ef83ce21f3701151421380e82f68e3b9fffefcd9042a89312334a5ab74dcf"} Nov 04 01:19:44.255589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:44.252012 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 04 01:19:44.255589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:44.252076 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" podUID="d26cecde-b16d-4fe1-add0-41a6d0a5301a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 04 01:19:44.335848 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:44.335811 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" event={"ID":"169d147a-6807-4836-8eca-4b7aa3d33bd5","Type":"ContainerStarted","Data":"aa8ecffc130835fd35082b1f1ff491a853a2d44bb45b907b64a30b00d6a40562"} Nov 04 01:19:44.384283 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:44.384211 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" podStartSLOduration=2.384185799 podStartE2EDuration="2.384185799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:44.319487115 +0000 UTC m=+1266.434597234" watchObservedRunningTime="2026-11-04 01:19:44.384185799 +0000 UTC m=+1266.499295918" Nov 04 01:19:44.583532 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:44.583473 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:44.593155 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:44.593125 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 01:19:44.593466 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:44.593414 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" podUID="36db7091-781e-4f96-a979-6ad25e0fb53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 01:19:44.724588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:44.721429 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:44.724588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:44.721642 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 04 01:19:44.724588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:44.721692 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" podUID="8c157ec5-7fa3-4774-bace-71cd27ee9d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 04 01:19:44.801587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:44.799172 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:44.801587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:44.799295 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 04 01:19:44.801587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:44.799342 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" podUID="169d147a-6807-4836-8eca-4b7aa3d33bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 04 01:19:45.257590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:45.250130 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 04 01:19:45.257590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:45.250207 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" podUID="d26cecde-b16d-4fe1-add0-41a6d0a5301a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 04 01:19:45.594213 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:45.594086 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 01:19:45.594517 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:45.594484 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" podUID="36db7091-781e-4f96-a979-6ad25e0fb53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 01:19:45.634472 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:45.634436 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:45.723586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:45.720030 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 04 01:19:45.723586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:45.720090 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" podUID="8c157ec5-7fa3-4774-bace-71cd27ee9d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 04 01:19:45.800590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:45.797939 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 04 01:19:45.800590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:45.798003 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" podUID="169d147a-6807-4836-8eca-4b7aa3d33bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 04 01:19:45.824589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:45.823636 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:19:45.829590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:45.826067 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:19:45.929659 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:45.927683 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:45.929659 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:45.928316 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:46.005597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:46.003710 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:46.050708 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:46.050665 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:46.251667 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:46.250020 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 04 01:19:46.251667 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:46.250084 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" podUID="d26cecde-b16d-4fe1-add0-41a6d0a5301a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 04 01:19:46.593587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:46.593040 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 01:19:46.593587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:46.593107 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" podUID="36db7091-781e-4f96-a979-6ad25e0fb53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 01:19:46.731591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:46.724090 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 04 01:19:46.731591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:46.724184 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" podUID="8c157ec5-7fa3-4774-bace-71cd27ee9d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 04 01:19:46.755605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:46.754653 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:19:46.787588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:46.786694 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:46.802589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:46.799927 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 04 01:19:46.802589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:46.799982 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" podUID="169d147a-6807-4836-8eca-4b7aa3d33bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 04 01:19:46.827591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:46.826677 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:19:46.827766 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:46.827679 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:19:46.849689 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:46.840456 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:46.870590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:46.850828 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" podStartSLOduration=4.850790816 podStartE2EDuration="4.850790816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:44.378976763 +0000 UTC m=+1266.494086884" watchObservedRunningTime="2026-11-04 01:19:46.850790816 +0000 UTC m=+1268.965900935" Nov 04 01:19:46.970591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:46.967362 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:46.997603 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:46.971964 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:19:47.013375 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:47.013335 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:47.014695 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:47.014669 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:19:47.074594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:47.052596 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:47.100644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:47.075350 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:47.279590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:47.278952 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 04 01:19:47.279590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:47.279013 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" podUID="d26cecde-b16d-4fe1-add0-41a6d0a5301a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 04 01:19:47.329588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:47.328814 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:47.333587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:47.332642 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": context deadline exceeded" Nov 04 01:19:47.593182 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:47.593145 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 01:19:47.593497 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:47.593449 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" podUID="36db7091-781e-4f96-a979-6ad25e0fb53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 01:19:47.642519 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:47.642490 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:19:47.732787 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:47.732739 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:47.759408 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:47.758630 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:47.759408 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:47.758999 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:19:47.774604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:47.774349 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 04 01:19:47.774604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:47.774402 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" podUID="8c157ec5-7fa3-4774-bace-71cd27ee9d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 04 01:19:47.791740 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:47.791700 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 04 01:19:47.792308 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:47.792291 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:19:47.913639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:47.910957 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 04 01:19:47.913639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:47.911017 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" podUID="169d147a-6807-4836-8eca-4b7aa3d33bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 04 01:19:47.913639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:47.912636 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:19:47.976867 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:47.976831 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:48.080807 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:48.080763 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:19:48.106997 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:48.091721 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:48.269646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:48.251951 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 04 01:19:48.269646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:48.252011 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" podUID="d26cecde-b16d-4fe1-add0-41a6d0a5301a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 04 01:19:48.303591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:48.294147 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:19:48.605755 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:48.600065 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 01:19:48.605755 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:48.600125 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" podUID="36db7091-781e-4f96-a979-6ad25e0fb53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 01:19:48.666054 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:48.666018 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:48.721266 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:48.721233 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 04 01:19:48.721530 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:48.721503 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" podUID="8c157ec5-7fa3-4774-bace-71cd27ee9d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 04 01:19:48.733683 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:48.733643 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:48.734263 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:48.734242 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:19:48.763149 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:48.763112 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:48.793794 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:48.793753 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:48.798330 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:48.798301 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 04 01:19:48.798624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:48.798541 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" podUID="169d147a-6807-4836-8eca-4b7aa3d33bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 04 01:19:48.940488 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:48.940380 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:48.963590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:48.951652 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:48.964596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:48.964336 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:19:48.983818 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:48.982726 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:49.144806 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:49.144759 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:19:49.184591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:49.170956 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:19:49.253697 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:49.253659 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 04 01:19:49.253864 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:49.253727 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" podUID="d26cecde-b16d-4fe1-add0-41a6d0a5301a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 04 01:19:49.327810 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:49.327760 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:49.348351 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:49.348310 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:19:49.455889 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:49.455856 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:19:49.594387 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:49.593910 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 01:19:49.594387 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:49.593971 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" podUID="36db7091-781e-4f96-a979-6ad25e0fb53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 01:19:49.716615 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:49.716583 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:49.734584 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:49.718238 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:49.735205 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:49.726474 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 04 01:19:49.735623 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:49.726515 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:49.735623 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:49.735386 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" podUID="8c157ec5-7fa3-4774-bace-71cd27ee9d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 04 01:19:49.735623 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:49.727157 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:19:49.797326 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:49.797290 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 04 01:19:49.801667 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:49.801629 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" podUID="169d147a-6807-4836-8eca-4b7aa3d33bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 04 01:19:49.948441 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:49.948340 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": context deadline exceeded" Nov 04 01:19:49.968938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:49.968900 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:50.098588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:50.090871 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:19:50.113598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:50.112869 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:19:50.253855 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:50.253823 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 04 01:19:50.254113 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:50.254083 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" podUID="d26cecde-b16d-4fe1-add0-41a6d0a5301a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 04 01:19:50.508642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:50.507306 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:50.593237 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:50.593207 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 01:19:50.593444 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:50.593422 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" podUID="36db7091-781e-4f96-a979-6ad25e0fb53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 01:19:50.719321 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:50.719288 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 04 01:19:50.723631 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:50.723594 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" podUID="8c157ec5-7fa3-4774-bace-71cd27ee9d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 04 01:19:51.030503 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:51.029172 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:51.032066 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:51.031901 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:51.033901 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:51.032886 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:19:51.256539 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:51.256503 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:51.256912 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:51.256889 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:51.257806 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:51.257771 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:19:51.451899 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:51.451814 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:51.597190 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:51.597161 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:51.597497 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:51.597467 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:51.598248 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:51.598224 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:19:51.723940 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:51.723871 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:51.724318 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:51.724142 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:51.724857 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:51.724841 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:19:52.230588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:52.226986 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:52.954843 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:52.954344 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:53.642169 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:53.642133 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:54.089816 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:54.089787 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:54.322305 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:54.322271 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:55.127586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:55.127185 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:55.728755 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:55.728679 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:56.497677 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:56.497641 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:56.602201 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:56.602167 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39bf412b-ph6mq"] Nov 04 01:19:56.604813 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:56.604798 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39bf412b-ph6mq" Nov 04 01:19:56.613793 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:56.613766 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39bf412b-ph6mq"] Nov 04 01:19:56.614055 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:56.614035 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:19:56.614690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:56.614672 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:19:56.697254 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:56.697224 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v8lf\" (UniqueName: \"kubernetes.io/projected/35a44cda-acce-49f1-865e-96af8e234fd0-kube-api-access-5v8lf\") pod \"maas-api-revoke-sub-39bf412b-ph6mq\" (UID: \"35a44cda-acce-49f1-865e-96af8e234fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39bf412b-ph6mq" Nov 04 01:19:56.697413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:56.697276 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a44cda-acce-49f1-865e-96af8e234fd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39bf412b-ph6mq\" (UID: \"35a44cda-acce-49f1-865e-96af8e234fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39bf412b-ph6mq" Nov 04 01:19:56.697413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:56.697355 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a44cda-acce-49f1-865e-96af8e234fd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39bf412b-ph6mq\" (UID: \"35a44cda-acce-49f1-865e-96af8e234fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39bf412b-ph6mq" Nov 04 01:19:56.798159 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:56.798123 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v8lf\" (UniqueName: \"kubernetes.io/projected/35a44cda-acce-49f1-865e-96af8e234fd0-kube-api-access-5v8lf\") pod \"maas-api-revoke-sub-39bf412b-ph6mq\" (UID: \"35a44cda-acce-49f1-865e-96af8e234fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39bf412b-ph6mq" Nov 04 01:19:56.798332 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:56.798191 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a44cda-acce-49f1-865e-96af8e234fd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39bf412b-ph6mq\" (UID: \"35a44cda-acce-49f1-865e-96af8e234fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39bf412b-ph6mq" Nov 04 01:19:56.798332 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:56.798234 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a44cda-acce-49f1-865e-96af8e234fd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39bf412b-ph6mq\" (UID: \"35a44cda-acce-49f1-865e-96af8e234fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39bf412b-ph6mq" Nov 04 01:19:56.799072 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:56.798989 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a44cda-acce-49f1-865e-96af8e234fd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39bf412b-ph6mq\" (UID: \"35a44cda-acce-49f1-865e-96af8e234fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39bf412b-ph6mq" Nov 04 01:19:56.800696 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:56.800676 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a44cda-acce-49f1-865e-96af8e234fd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39bf412b-ph6mq\" (UID: \"35a44cda-acce-49f1-865e-96af8e234fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39bf412b-ph6mq" Nov 04 01:19:56.807648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:56.807629 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v8lf\" (UniqueName: \"kubernetes.io/projected/35a44cda-acce-49f1-865e-96af8e234fd0-kube-api-access-5v8lf\") pod \"maas-api-revoke-sub-39bf412b-ph6mq\" (UID: \"35a44cda-acce-49f1-865e-96af8e234fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39bf412b-ph6mq" Nov 04 01:19:56.918786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:56.918749 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39bf412b-ph6mq" Nov 04 01:19:57.196827 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.196799 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39bf412b-ph6mq"] Nov 04 01:19:57.199774 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:19:57.199470 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a44cda_acce_49f1_865e_96af8e234fd0.slice/crio-cf50b22dbcca28df1ab1ad9dc37e438065c910f648d2cdb781f43e136580332b WatchSource:0}: Error finding container cf50b22dbcca28df1ab1ad9dc37e438065c910f648d2cdb781f43e136580332b: Status 404 returned error can't find the container with id cf50b22dbcca28df1ab1ad9dc37e438065c910f648d2cdb781f43e136580332b Nov 04 01:19:57.200810 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.200782 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p"] Nov 04 01:19:57.202368 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.202187 4852 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:19:57.205298 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.205274 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.241619 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.241588 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p"] Nov 04 01:19:57.302128 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.302096 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-099dd9-openshift-default-dockercfg-ntzzj\"" Nov 04 01:19:57.304333 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.303848 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.304333 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.303890 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.304333 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.303963 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.304333 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.304038 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.304333 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.304158 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.304333 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.304194 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.306647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.304393 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkcjp\" (UniqueName: \"kubernetes.io/projected/baefa3c0-28be-428a-9591-f0ca1f07fafa-kube-api-access-wkcjp\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.306647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.304441 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baefa3c0-28be-428a-9591-f0ca1f07fafa-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.306647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.304475 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.405879 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.405850 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baefa3c0-28be-428a-9591-f0ca1f07fafa-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.406036 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.405887 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.406036 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.405939 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.406036 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.405954 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.406036 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.405990 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.406036 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.406029 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.406301 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.406060 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.406301 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.406082 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.406401 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.406337 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.406401 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.406384 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkcjp\" (UniqueName: \"kubernetes.io/projected/baefa3c0-28be-428a-9591-f0ca1f07fafa-kube-api-access-wkcjp\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.406501 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.406471 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.406611 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.406588 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.406611 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.406597 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.406746 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.406666 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baefa3c0-28be-428a-9591-f0ca1f07fafa-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.408648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.408622 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.408648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.408642 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.425028 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.425006 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.425264 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.425248 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkcjp\" (UniqueName: \"kubernetes.io/projected/baefa3c0-28be-428a-9591-f0ca1f07fafa-kube-api-access-wkcjp\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.433293 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.433267 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:57.450071 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.450009 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl"] Nov 04 01:19:57.454156 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.454134 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.469609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.469584 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl"] Nov 04 01:19:57.507483 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.507449 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sw5d\" (UniqueName: \"kubernetes.io/projected/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-kube-api-access-4sw5d\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.507982 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.507638 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.507982 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.507710 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.507982 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.507730 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.507982 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.507788 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.507982 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.507819 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.507982 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.507851 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.507982 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.507967 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.508362 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.508002 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.524413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.524119 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:57.551665 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.551636 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ee876a-openshift-default-dockercfg-v2pll\"" Nov 04 01:19:57.622591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.610662 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.622591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.610716 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.622591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.610758 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.622591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.610797 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.622591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.610827 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.622591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.610875 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sw5d\" (UniqueName: \"kubernetes.io/projected/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-kube-api-access-4sw5d\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.622591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.611008 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.622591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.611046 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.622591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.611076 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.622591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.612133 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.622591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.613403 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.622591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.613706 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.622591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.613953 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.622591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.614211 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.622591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.615396 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.622591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.617823 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.622591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.621833 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sw5d\" (UniqueName: \"kubernetes.io/projected/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-kube-api-access-4sw5d\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.624321 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.624282 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.624517 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.624479 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39bf412b-ph6mq" event={"ID":"35a44cda-acce-49f1-865e-96af8e234fd0","Type":"ContainerStarted","Data":"cf50b22dbcca28df1ab1ad9dc37e438065c910f648d2cdb781f43e136580332b"} Nov 04 01:19:57.776716 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.776687 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:57.809374 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.809347 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p"] Nov 04 01:19:57.811694 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:19:57.811667 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaefa3c0_28be_428a_9591_f0ca1f07fafa.slice/crio-8fdd53a1e689ddedbdaf659d7f8018bf5054b8ce95641113e49102c8880f4f03 WatchSource:0}: Error finding container 8fdd53a1e689ddedbdaf659d7f8018bf5054b8ce95641113e49102c8880f4f03: Status 404 returned error can't find the container with id 8fdd53a1e689ddedbdaf659d7f8018bf5054b8ce95641113e49102c8880f4f03 Nov 04 01:19:57.814428 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.814392 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:57.814523 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.814460 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:57.814523 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:57.814490 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:58.059945 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:58.059908 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl"] Nov 04 01:19:58.060862 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:19:58.060835 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a4d4cee_bd81_4a99_8d45_c636a02c64cf.slice/crio-eaea6aad4e9be4307e03033629195395ca4d649b845ef8b0fa12fa4e68c4b6d1 WatchSource:0}: Error finding container eaea6aad4e9be4307e03033629195395ca4d649b845ef8b0fa12fa4e68c4b6d1: Status 404 returned error can't find the container with id eaea6aad4e9be4307e03033629195395ca4d649b845ef8b0fa12fa4e68c4b6d1 Nov 04 01:19:58.063489 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:58.063454 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:58.063650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:58.063526 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:58.063650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:58.063558 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:58.136836 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:58.136760 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:58.633586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:58.633526 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" event={"ID":"baefa3c0-28be-428a-9591-f0ca1f07fafa","Type":"ContainerStarted","Data":"b8bce3eaa168e4e58088218699c821309762b5813ac17399f4653bec114d0e74"} Nov 04 01:19:58.634082 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:58.633626 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" event={"ID":"baefa3c0-28be-428a-9591-f0ca1f07fafa","Type":"ContainerStarted","Data":"8fdd53a1e689ddedbdaf659d7f8018bf5054b8ce95641113e49102c8880f4f03"} Nov 04 01:19:58.635989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:58.635955 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" event={"ID":"3a4d4cee-bd81-4a99-8d45-c636a02c64cf","Type":"ContainerStarted","Data":"435f0eb11db1abdba93331fd4fcda004baef191174a6fd5edfc3d881c235b2c4"} Nov 04 01:19:58.636161 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:58.636013 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" event={"ID":"3a4d4cee-bd81-4a99-8d45-c636a02c64cf","Type":"ContainerStarted","Data":"eaea6aad4e9be4307e03033629195395ca4d649b845ef8b0fa12fa4e68c4b6d1"} Nov 04 01:19:58.668591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:58.656833 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" podStartSLOduration=1.6568151389999999 podStartE2EDuration="1.656815139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:58.655856175 +0000 UTC m=+1280.770966295" watchObservedRunningTime="2026-11-04 01:19:58.656815139 +0000 UTC m=+1280.771925260" Nov 04 01:19:58.687588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:58.683311 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" podStartSLOduration=1.6832918220000002 podStartE2EDuration="1.683291822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:58.682114076 +0000 UTC m=+1280.797224195" watchObservedRunningTime="2026-11-04 01:19:58.683291822 +0000 UTC m=+1280.798401940" Nov 04 01:19:58.801504 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:58.800671 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:19:59.075166 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:59.075077 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:19:59.524883 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:59.524848 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:19:59.526237 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:59.526208 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 04 01:19:59.526388 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:59.526261 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" podUID="baefa3c0-28be-428a-9591-f0ca1f07fafa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 04 01:19:59.812650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:59.805190 4852 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 04 01:19:59.812650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:19:59.805249 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" podUID="3a4d4cee-bd81-4a99-8d45-c636a02c64cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 04 01:19:59.916829 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:19:59.916790 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:20:00.042025 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:00.040665 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:00.524653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:00.524609 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 04 01:20:00.524824 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:00.524680 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" podUID="baefa3c0-28be-428a-9591-f0ca1f07fafa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 04 01:20:00.787532 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:00.787013 4852 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 04 01:20:00.787532 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:00.787082 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" podUID="3a4d4cee-bd81-4a99-8d45-c636a02c64cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 04 01:20:00.826949 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:00.826913 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:01.525478 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:01.525438 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 04 01:20:01.525795 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:01.525498 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" podUID="baefa3c0-28be-428a-9591-f0ca1f07fafa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 04 01:20:01.586603 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:01.580442 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:01.809653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:01.809283 4852 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 04 01:20:01.809653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:01.809381 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" podUID="3a4d4cee-bd81-4a99-8d45-c636a02c64cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 04 01:20:02.533594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:02.531534 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 04 01:20:02.533594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:02.531633 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" podUID="baefa3c0-28be-428a-9591-f0ca1f07fafa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 04 01:20:02.556263 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:02.555864 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:02.785512 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:02.785037 4852 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 04 01:20:02.785512 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:02.785107 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" podUID="3a4d4cee-bd81-4a99-8d45-c636a02c64cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 04 01:20:03.527033 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:03.526473 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 04 01:20:03.527033 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:03.526549 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" podUID="baefa3c0-28be-428a-9591-f0ca1f07fafa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 04 01:20:03.528665 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:03.528536 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:03.756929 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:03.751220 4852 generic.go:356] "Generic (PLEG): container finished" podID="35a44cda-acce-49f1-865e-96af8e234fd0" containerID="702e658f6afb66140266c4a22eabea4bb7b0ca67dfbb476045ec386d1cef9929" exitCode=0 Nov 04 01:20:03.756929 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:03.751273 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39bf412b-ph6mq" event={"ID":"35a44cda-acce-49f1-865e-96af8e234fd0","Type":"ContainerDied","Data":"702e658f6afb66140266c4a22eabea4bb7b0ca67dfbb476045ec386d1cef9929"} Nov 04 01:20:03.779526 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:03.779494 4852 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 04 01:20:03.779868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:03.779837 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" podUID="3a4d4cee-bd81-4a99-8d45-c636a02c64cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 04 01:20:04.428589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:04.425388 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:04.526927 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:04.526887 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 04 01:20:04.527146 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:04.526963 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" podUID="baefa3c0-28be-428a-9591-f0ca1f07fafa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 04 01:20:04.777454 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:04.777417 4852 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 04 01:20:04.777454 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:04.777470 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" podUID="3a4d4cee-bd81-4a99-8d45-c636a02c64cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 04 01:20:04.918974 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:04.918941 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39bf412b-ph6mq" Nov 04 01:20:05.080215 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:05.080123 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35a44cda-acce-49f1-865e-96af8e234fd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a44cda-acce-49f1-865e-96af8e234fd0-maas-api-openshift-ca\") pod \"35a44cda-acce-49f1-865e-96af8e234fd0\" (UID: \"35a44cda-acce-49f1-865e-96af8e234fd0\") " Nov 04 01:20:05.080215 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:05.080187 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35a44cda-acce-49f1-865e-96af8e234fd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a44cda-acce-49f1-865e-96af8e234fd0-maas-api-xks-ca\") pod \"35a44cda-acce-49f1-865e-96af8e234fd0\" (UID: \"35a44cda-acce-49f1-865e-96af8e234fd0\") " Nov 04 01:20:05.080447 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:05.080226 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a44cda-acce-49f1-865e-96af8e234fd0-kube-api-access-5v8lf\" (UniqueName: \"kubernetes.io/projected/35a44cda-acce-49f1-865e-96af8e234fd0-kube-api-access-5v8lf\") pod \"35a44cda-acce-49f1-865e-96af8e234fd0\" (UID: \"35a44cda-acce-49f1-865e-96af8e234fd0\") " Nov 04 01:20:05.080658 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:05.080557 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a44cda-acce-49f1-865e-96af8e234fd0-maas-api-openshift-ca" pod "35a44cda-acce-49f1-865e-96af8e234fd0" (UID: "35a44cda-acce-49f1-865e-96af8e234fd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:05.082966 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:05.082919 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a44cda-acce-49f1-865e-96af8e234fd0-kube-api-access-5v8lf" pod "35a44cda-acce-49f1-865e-96af8e234fd0" (UID: "35a44cda-acce-49f1-865e-96af8e234fd0"). InnerVolumeSpecName "kube-api-access-5v8lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:05.083250 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:05.083226 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a44cda-acce-49f1-865e-96af8e234fd0-maas-api-xks-ca" pod "35a44cda-acce-49f1-865e-96af8e234fd0" (UID: "35a44cda-acce-49f1-865e-96af8e234fd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:05.182415 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:05.182368 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a44cda-acce-49f1-865e-96af8e234fd0-maas-api-xks-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:05.182415 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:05.182413 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v8lf\" (UniqueName: \"kubernetes.io/projected/35a44cda-acce-49f1-865e-96af8e234fd0-kube-api-access-5v8lf\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:05.182743 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:05.182429 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a44cda-acce-49f1-865e-96af8e234fd0-maas-api-openshift-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:05.212167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:05.210782 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:05.529724 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:05.529691 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:20:05.530008 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:05.529988 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:20:05.530944 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:05.530923 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:20:05.765395 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:05.765367 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39bf412b-ph6mq" Nov 04 01:20:05.765612 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:05.765403 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39bf412b-ph6mq" event={"ID":"35a44cda-acce-49f1-865e-96af8e234fd0","Type":"ContainerDied","Data":"cf50b22dbcca28df1ab1ad9dc37e438065c910f648d2cdb781f43e136580332b"} Nov 04 01:20:05.765612 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:05.765465 4852 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf50b22dbcca28df1ab1ad9dc37e438065c910f648d2cdb781f43e136580332b" Nov 04 01:20:06.026342 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:06.026306 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:06.294899 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:06.294815 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:20:06.295184 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:06.295154 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:20:06.296203 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:06.296176 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:20:06.724063 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:06.723974 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:07.535324 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:07.535290 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:08.335068 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:08.335033 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:09.139395 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:09.139354 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:10.534317 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:10.534268 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:11.523885 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:11.523833 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:12.261878 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:12.261847 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:13.034185 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:13.034144 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:13.766940 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:13.766278 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:14.566639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:14.566603 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:15.416797 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:15.416727 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:16.261910 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:16.261857 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:17.725588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:17.721121 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:18.846712 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:18.846671 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:20.032584 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:20.032507 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:20.726589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:20.723460 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:24.104333 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:24.104295 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:24.253310 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:24.253277 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:24.770907 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:24.765284 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:25.443238 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:25.443085 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:26.308574 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:26.308530 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:26.955632 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:26.953249 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:27.002016 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.001980 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr"] Nov 04 01:20:27.002469 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.002454 4852 cpu_manager.go:414] "containerMap: removing container" podUID="35a44cda-acce-49f1-865e-96af8e234fd0" containerName="revoke-keys" Nov 04 01:20:27.002521 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.002470 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a44cda-acce-49f1-865e-96af8e234fd0" containerName="revoke-keys" Nov 04 01:20:27.002610 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.002579 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a44cda-acce-49f1-865e-96af8e234fd0" containerName="revoke-keys" Nov 04 01:20:27.005825 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.005803 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.021981 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.021949 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr"] Nov 04 01:20:27.053061 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.053024 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7619e54f-752b-428f-b955-b2dec4471c5b-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.053296 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.053080 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.053296 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.053231 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.053296 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.053288 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.053532 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.053505 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.053686 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.053636 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7619e54f-752b-428f-b955-b2dec4471c5b-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.053686 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.053681 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7619e54f-752b-428f-b955-b2dec4471c5b-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.053850 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.053825 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.053963 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.053947 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-965lz\" (UniqueName: \"kubernetes.io/projected/7619e54f-752b-428f-b955-b2dec4471c5b-kube-api-access-965lz\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.154840 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.154806 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.154840 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.154848 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.155122 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.154872 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.155122 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.155005 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7619e54f-752b-428f-b955-b2dec4471c5b-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.155122 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.155068 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7619e54f-752b-428f-b955-b2dec4471c5b-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.155277 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.155128 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.155277 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.155197 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-965lz\" (UniqueName: \"kubernetes.io/projected/7619e54f-752b-428f-b955-b2dec4471c5b-kube-api-access-965lz\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.155277 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.155262 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.155433 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.155288 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7619e54f-752b-428f-b955-b2dec4471c5b-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.155433 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.155325 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.155433 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.155351 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.155607 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.155512 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.155851 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.155829 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7619e54f-752b-428f-b955-b2dec4471c5b-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.155964 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.155946 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.158030 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.158009 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.158242 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.158222 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7619e54f-752b-428f-b955-b2dec4471c5b-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.164271 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.164220 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7619e54f-752b-428f-b955-b2dec4471c5b-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.164271 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.164257 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-965lz\" (UniqueName: \"kubernetes.io/projected/7619e54f-752b-428f-b955-b2dec4471c5b-kube-api-access-965lz\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.319987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.319951 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:27.612532 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.612505 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr"] Nov 04 01:20:27.614760 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:20:27.614724 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7619e54f_752b_428f_b955_b2dec4471c5b.slice/crio-ee9904149a8a9ffb9c32a62ae8419c82a203d6f8b372329bc026fc83831b065b WatchSource:0}: Error finding container ee9904149a8a9ffb9c32a62ae8419c82a203d6f8b372329bc026fc83831b065b: Status 404 returned error can't find the container with id ee9904149a8a9ffb9c32a62ae8419c82a203d6f8b372329bc026fc83831b065b Nov 04 01:20:27.617780 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.617742 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:27.617894 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.617832 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:27.617894 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.617870 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:27.757662 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.752008 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:27.996511 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.996427 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" event={"ID":"7619e54f-752b-428f-b955-b2dec4471c5b","Type":"ContainerStarted","Data":"7b84e0ed4b23e5e32b42ee9aa15b1acf044ce593c365273c5472ff267e6dc873"} Nov 04 01:20:27.996511 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:27.996486 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" event={"ID":"7619e54f-752b-428f-b955-b2dec4471c5b","Type":"ContainerStarted","Data":"ee9904149a8a9ffb9c32a62ae8419c82a203d6f8b372329bc026fc83831b065b"} Nov 04 01:20:28.028323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:28.028266 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" podStartSLOduration=2.028251622 podStartE2EDuration="2.028251622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:28.026704399 +0000 UTC m=+1310.141814514" watchObservedRunningTime="2026-11-04 01:20:28.028251622 +0000 UTC m=+1310.143361802" Nov 04 01:20:28.320917 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:28.320869 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:28.322913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:28.322855 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:20:28.323263 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:28.323230 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" podUID="7619e54f-752b-428f-b955-b2dec4471c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 04 01:20:28.561594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:28.556988 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:29.002245 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:29.002211 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:29.320650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:29.320608 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:20:29.320853 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:29.320673 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" podUID="7619e54f-752b-428f-b955-b2dec4471c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 04 01:20:29.574145 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:29.574054 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:30.120952 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:20:30.120907 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:20:30.321154 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:30.321117 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:20:30.321492 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:30.321459 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" podUID="7619e54f-752b-428f-b955-b2dec4471c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 04 01:20:30.644777 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:30.644715 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:20:30.686784 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:30.686721 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:30.842760 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:30.842704 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:30.915669 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:30.912844 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:30.996712 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:30.996667 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:20:31.242685 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:31.238467 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:31.323387 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:31.322941 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:20:31.323387 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:31.323034 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" podUID="7619e54f-752b-428f-b955-b2dec4471c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 04 01:20:32.320383 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:32.320347 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:20:32.320837 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:32.320401 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" podUID="7619e54f-752b-428f-b955-b2dec4471c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 04 01:20:32.471093 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:32.471029 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:32.681774 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:32.681664 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:33.320672 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:33.320636 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:20:33.321077 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:33.320692 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" podUID="7619e54f-752b-428f-b955-b2dec4471c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 04 01:20:33.427609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:33.427523 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:34.320622 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:34.320584 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:20:34.320853 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:34.320659 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" podUID="7619e54f-752b-428f-b955-b2dec4471c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 04 01:20:34.369430 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:34.369386 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:35.132591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:35.132383 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:35.891140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:35.891107 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:36.083416 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:36.083380 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:36.083920 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:36.083889 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:36.085182 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:36.085161 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:20:36.827677 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:36.827640 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:37.646333 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:37.646269 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:38.542587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:38.538115 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:39.212622 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:39.212170 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:40.005586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:40.003091 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:40.285045 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:40.284950 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:40.857588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:40.855172 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:41.342553 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:41.342510 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:41.493604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:41.493471 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:41.672436 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:41.672351 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:42.046931 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:42.046898 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:42.368638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:42.367043 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:42.908602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:42.908554 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:43.235145 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:43.235059 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:43.304345 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:43.304313 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:20:44.164461 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:44.164420 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:45.713228 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:45.713196 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:46.537056 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:46.537014 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:47.265170 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:47.265134 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:48.023663 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:48.023620 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:48.817016 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:48.816980 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:49.640052 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:49.639992 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:50.158607 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:50.151113 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:50.530234 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:50.530168 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:51.246424 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:51.246390 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:51.942029 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:51.941992 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:52.718238 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:52.718205 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:53.623135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:53.623096 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:54.437588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:54.434062 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:55.187591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:55.180962 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:55.824513 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:55.824477 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:55.994543 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:55.994510 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:56.508175 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:56.508141 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:57.220862 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.220782 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:57.282128 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.282052 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw"] Nov 04 01:20:57.286286 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.286257 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.299810 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.299738 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw"] Nov 04 01:20:57.365870 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.365834 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.366042 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.365986 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.366042 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.366023 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79qks\" (UniqueName: \"kubernetes.io/projected/ab9122ac-8874-4a64-9137-a7fe056db11e-kube-api-access-79qks\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.366203 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.366172 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.366203 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.366215 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9122ac-8874-4a64-9137-a7fe056db11e-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.366420 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.366322 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.366420 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.366362 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.366420 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.366387 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.366702 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.366485 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.467551 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.467510 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.467768 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.467604 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.467768 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.467694 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.467768 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.467725 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79qks\" (UniqueName: \"kubernetes.io/projected/ab9122ac-8874-4a64-9137-a7fe056db11e-kube-api-access-79qks\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.467942 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.467779 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.467942 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.467805 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9122ac-8874-4a64-9137-a7fe056db11e-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.467942 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.467851 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.467942 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.467886 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.467942 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.467909 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.468208 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.467958 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.468272 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.468236 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.468635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.468602 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.469217 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.469028 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.469217 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.469100 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9122ac-8874-4a64-9137-a7fe056db11e-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.470684 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.470660 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.471328 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.471274 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.476490 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.476464 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.477363 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.477342 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79qks\" (UniqueName: \"kubernetes.io/projected/ab9122ac-8874-4a64-9137-a7fe056db11e-kube-api-access-79qks\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.507251 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.507230 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx"] Nov 04 01:20:57.512289 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.512273 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.524647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.524595 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx"] Nov 04 01:20:57.569299 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.569269 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.569451 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.569311 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.569451 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.569332 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.569451 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.569440 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.569667 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.569481 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.569667 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.569543 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.569667 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.569611 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.569667 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.569628 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.569875 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.569676 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgzsb\" (UniqueName: \"kubernetes.io/projected/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-kube-api-access-bgzsb\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.598493 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.598467 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx"] Nov 04 01:20:57.602747 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.602729 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.610259 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.610220 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:57.613393 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.612956 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx"] Nov 04 01:20:57.672503 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.672466 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz"] Nov 04 01:20:57.673827 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.673794 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.673974 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.673854 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlghl\" (UniqueName: \"kubernetes.io/projected/a5c642db-a1e8-4839-9fa7-1821588ecb15-kube-api-access-mlghl\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.673974 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.673891 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.673974 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.673917 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.673974 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.673951 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.673974 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.673975 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.674257 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.673989 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.674257 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.674031 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgzsb\" (UniqueName: \"kubernetes.io/projected/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-kube-api-access-bgzsb\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.674257 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.674058 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.674257 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.674091 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c642db-a1e8-4839-9fa7-1821588ecb15-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.674257 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.674132 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.674257 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.674182 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.674257 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.674209 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.674257 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.674240 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.674700 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.674277 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.674700 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.674298 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.674700 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.674317 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.674700 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.674336 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.674908 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.674713 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.677392 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.675725 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.677392 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.676041 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.677392 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.676674 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.677392 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.676905 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.677392 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.677304 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.679136 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.679120 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.701266 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.697029 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.701266 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.697374 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.701266 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.700414 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz"] Nov 04 01:20:57.703763 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.703740 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgzsb\" (UniqueName: \"kubernetes.io/projected/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-kube-api-access-bgzsb\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.775868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.775841 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.776022 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.775873 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b03d6b4-7003-4705-b3df-64461213d3bc-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.776022 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.775962 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.776152 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.776068 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wglch\" (UniqueName: \"kubernetes.io/projected/6b03d6b4-7003-4705-b3df-64461213d3bc-kube-api-access-wglch\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.776152 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.776093 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.776152 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.776124 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.776369 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.776168 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.776369 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.776240 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlghl\" (UniqueName: \"kubernetes.io/projected/a5c642db-a1e8-4839-9fa7-1821588ecb15-kube-api-access-mlghl\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.776369 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.776278 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.776369 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.776359 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.776598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.776388 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.776598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.776393 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.776598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.776441 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.776598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.776468 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.776598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.776499 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.776598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.776540 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c642db-a1e8-4839-9fa7-1821588ecb15-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.776921 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.776628 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.776921 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.776679 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.776921 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.776720 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.776921 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.776756 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.777136 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.776930 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.777136 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.776995 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.777506 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.777481 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c642db-a1e8-4839-9fa7-1821588ecb15-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.779384 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.779359 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.779464 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.779450 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.784896 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.784875 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.785087 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.785071 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlghl\" (UniqueName: \"kubernetes.io/projected/a5c642db-a1e8-4839-9fa7-1821588ecb15-kube-api-access-mlghl\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.844239 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.844219 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:57.877929 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.877905 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.878065 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.877958 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.878065 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.877999 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b03d6b4-7003-4705-b3df-64461213d3bc-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.878065 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.878041 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wglch\" (UniqueName: \"kubernetes.io/projected/6b03d6b4-7003-4705-b3df-64461213d3bc-kube-api-access-wglch\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.878226 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.878194 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.878282 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.878237 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.878282 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.878266 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.878385 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.878294 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.878385 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.878321 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.878385 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.878359 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.878816 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.878774 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.879020 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.878987 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.879104 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.879023 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.879104 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.879073 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b03d6b4-7003-4705-b3df-64461213d3bc-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.880957 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.880935 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.881853 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.881832 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.889366 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.889293 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.890508 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.890486 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wglch\" (UniqueName: \"kubernetes.io/projected/6b03d6b4-7003-4705-b3df-64461213d3bc-kube-api-access-wglch\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:57.920102 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.920070 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:57.945787 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.943441 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw"] Nov 04 01:20:57.948459 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.948413 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:57.948651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.948552 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:57.948772 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:57.948748 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:58.023379 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.023345 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:20:58.132972 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.132939 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx"] Nov 04 01:20:58.137617 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:20:58.137531 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719d829c_c70a_4bdc_a3d2_5b4cb52a45da.slice/crio-804f4a4597786e809539f92397e46d3fa65b3450a47a8a17f7690887fb30f962 WatchSource:0}: Error finding container 804f4a4597786e809539f92397e46d3fa65b3450a47a8a17f7690887fb30f962: Status 404 returned error can't find the container with id 804f4a4597786e809539f92397e46d3fa65b3450a47a8a17f7690887fb30f962 Nov 04 01:20:58.141646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.141602 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:58.141761 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.141696 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:58.141761 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.141738 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:58.257490 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.244536 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:58.288847 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.288682 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx"] Nov 04 01:20:58.293502 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:20:58.293479 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c642db_a1e8_4839_9fa7_1821588ecb15.slice/crio-5ab4e11a171abb7a1b2c91c0fb4eb011e54bf46af1152bc1051b69f234b3fa65 WatchSource:0}: Error finding container 5ab4e11a171abb7a1b2c91c0fb4eb011e54bf46af1152bc1051b69f234b3fa65: Status 404 returned error can't find the container with id 5ab4e11a171abb7a1b2c91c0fb4eb011e54bf46af1152bc1051b69f234b3fa65 Nov 04 01:20:58.296908 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.296268 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:58.296908 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.296347 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:58.296908 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.296385 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:58.339497 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.339410 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz"] Nov 04 01:20:58.351938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.350273 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:58.351938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.350365 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:58.351938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.350410 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:58.392763 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.392591 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" event={"ID":"719d829c-c70a-4bdc-a3d2-5b4cb52a45da","Type":"ContainerStarted","Data":"d1ad05404360bac0238027aa3d12aeff93e4b892302fa5a0de5acc3c1c2ac70b"} Nov 04 01:20:58.392763 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.392658 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" event={"ID":"719d829c-c70a-4bdc-a3d2-5b4cb52a45da","Type":"ContainerStarted","Data":"804f4a4597786e809539f92397e46d3fa65b3450a47a8a17f7690887fb30f962"} Nov 04 01:20:58.397093 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.397029 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" event={"ID":"a5c642db-a1e8-4839-9fa7-1821588ecb15","Type":"ContainerStarted","Data":"5ab4e11a171abb7a1b2c91c0fb4eb011e54bf46af1152bc1051b69f234b3fa65"} Nov 04 01:20:58.399009 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.398955 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" event={"ID":"6b03d6b4-7003-4705-b3df-64461213d3bc","Type":"ContainerStarted","Data":"2a1eb551c08e9a183f3b3438888f86f3e506efed827cbdfd900cbced43c3ab85"} Nov 04 01:20:58.403589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.402846 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" event={"ID":"ab9122ac-8874-4a64-9137-a7fe056db11e","Type":"ContainerStarted","Data":"341c1ef2ada34d68609175467758b55c188435bbdf83814e34ab1401e6866575"} Nov 04 01:20:58.403589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.402898 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" event={"ID":"ab9122ac-8874-4a64-9137-a7fe056db11e","Type":"ContainerStarted","Data":"89930da40acbb8a0dd573c488ef91a926e1127d227773b8dc5929229d1b4386e"} Nov 04 01:20:58.427017 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.425196 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" podStartSLOduration=1.425178277 podStartE2EDuration="1.425178277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:58.421986972 +0000 UTC m=+1340.537097095" watchObservedRunningTime="2026-11-04 01:20:58.425178277 +0000 UTC m=+1340.540288395" Nov 04 01:20:58.485072 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.466192 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" podStartSLOduration=1.466175316 podStartE2EDuration="1.466175316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:58.465925411 +0000 UTC m=+1340.581035530" watchObservedRunningTime="2026-11-04 01:20:58.466175316 +0000 UTC m=+1340.581285438" Nov 04 01:20:58.614704 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.611663 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:20:58.844908 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:58.844872 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:20:59.342216 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:59.340363 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:20:59.650966 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:59.650881 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 04 01:20:59.678725 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:59.678683 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" podUID="ab9122ac-8874-4a64-9137-a7fe056db11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 04 01:20:59.756901 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:59.713478 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" event={"ID":"a5c642db-a1e8-4839-9fa7-1821588ecb15","Type":"ContainerStarted","Data":"2d69104a66c37ac0deb4b0c03e18376f90b5eb0f18c4be7182a77d2f6ee4039a"} Nov 04 01:20:59.780656 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:59.780616 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" event={"ID":"6b03d6b4-7003-4705-b3df-64461213d3bc","Type":"ContainerStarted","Data":"7cf793fb503d17ce8969dad595474c0b451a20e88b8c69024d5c2e7264531b9e"} Nov 04 01:20:59.827863 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:59.827785 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" podStartSLOduration=2.827765025 podStartE2EDuration="2.827765025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:59.751647024 +0000 UTC m=+1341.866757144" watchObservedRunningTime="2026-11-04 01:20:59.827765025 +0000 UTC m=+1341.942875145" Nov 04 01:20:59.828644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:59.828601 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" podStartSLOduration=2.8285869740000003 podStartE2EDuration="2.828586974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:59.818999882 +0000 UTC m=+1341.934110001" watchObservedRunningTime="2026-11-04 01:20:59.828586974 +0000 UTC m=+1341.943697084" Nov 04 01:20:59.865665 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:59.861725 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 01:20:59.865665 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:59.861781 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" podUID="719d829c-c70a-4bdc-a3d2-5b4cb52a45da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 01:20:59.928104 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:59.923287 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:20:59.928905 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:59.928369 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 04 01:20:59.928905 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:20:59.928421 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" podUID="a5c642db-a1e8-4839-9fa7-1821588ecb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 04 01:20:59.971353 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:20:59.971316 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:21:00.045587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:00.026218 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:21:00.045587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:00.026350 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 04 01:21:00.045587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:00.026392 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" podUID="6b03d6b4-7003-4705-b3df-64461213d3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 04 01:21:00.437151 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:00.437119 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:00.613690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:00.613357 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 04 01:21:00.613690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:00.613412 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" podUID="ab9122ac-8874-4a64-9137-a7fe056db11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 04 01:21:00.692909 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:00.691683 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:00.729833 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:00.728671 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:00.848600 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:00.847964 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 01:21:00.848600 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:00.848024 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" podUID="719d829c-c70a-4bdc-a3d2-5b4cb52a45da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 01:21:00.932601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:00.926947 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 04 01:21:00.932601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:00.927008 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" podUID="a5c642db-a1e8-4839-9fa7-1821588ecb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 04 01:21:01.025024 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:01.024983 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 04 01:21:01.037509 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:01.036988 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" podUID="6b03d6b4-7003-4705-b3df-64461213d3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 04 01:21:01.630037 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:01.630000 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 04 01:21:01.664623 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:01.638713 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" podUID="ab9122ac-8874-4a64-9137-a7fe056db11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 04 01:21:01.935646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:01.931812 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 01:21:01.935646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:01.931911 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" podUID="719d829c-c70a-4bdc-a3d2-5b4cb52a45da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 01:21:01.935646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:01.932624 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 04 01:21:01.935646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:01.932664 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" podUID="a5c642db-a1e8-4839-9fa7-1821588ecb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 04 01:21:01.972590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:01.967681 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:02.017590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:01.989354 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:21:02.017590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:01.990510 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:02.083586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:02.079728 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 04 01:21:02.083586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:02.079789 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" podUID="6b03d6b4-7003-4705-b3df-64461213d3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 04 01:21:02.115032 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:02.113685 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:02.373053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:02.372993 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:02.427727 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:02.427676 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": context deadline exceeded" Nov 04 01:21:02.613642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:02.613601 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 04 01:21:02.613946 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:02.613901 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" podUID="ab9122ac-8874-4a64-9137-a7fe056db11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 04 01:21:02.666725 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:02.666675 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:02.691696 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:02.691652 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:02.795077 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:02.794201 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:02.864952 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:02.864909 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 04 01:21:02.872592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:02.868536 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 01:21:02.872831 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:02.872803 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" podUID="719d829c-c70a-4bdc-a3d2-5b4cb52a45da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 01:21:02.922136 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:02.922035 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 04 01:21:02.922388 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:02.922343 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" podUID="a5c642db-a1e8-4839-9fa7-1821588ecb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 04 01:21:02.934289 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:02.934180 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:02.944428 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:02.943653 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:02.995788 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:02.995729 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:02.996504 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:02.996483 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:21:02.997966 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:02.997942 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:02.998423 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:02.998400 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:21:03.042682 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:03.042647 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 04 01:21:03.051754 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:03.051672 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" podUID="6b03d6b4-7003-4705-b3df-64461213d3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 04 01:21:03.117727 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:03.117678 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:03.213209 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:03.143027 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:21:03.387705 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:03.387652 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:03.388710 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:03.388686 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:21:03.622993 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:03.622955 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 04 01:21:03.623378 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:03.623349 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" podUID="ab9122ac-8874-4a64-9137-a7fe056db11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 04 01:21:03.669700 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:03.669644 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:21:03.675588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:03.671514 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:21:03.692782 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:03.692747 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 04 01:21:03.693507 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:03.693484 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:21:03.729590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:03.729322 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:03.814699 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:03.807858 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:03.854596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:03.850586 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 01:21:03.858613 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:03.854909 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" podUID="719d829c-c70a-4bdc-a3d2-5b4cb52a45da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 01:21:03.936466 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:03.936366 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 04 01:21:03.936777 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:03.936730 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" podUID="a5c642db-a1e8-4839-9fa7-1821588ecb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 04 01:21:03.945771 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:03.945728 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:03.957462 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:03.957273 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:21:03.996411 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:03.996371 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:21:04.009868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:04.009829 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:04.026596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:04.021133 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:04.027308 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:04.027283 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 04 01:21:04.027485 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:04.027459 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" podUID="6b03d6b4-7003-4705-b3df-64461213d3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 04 01:21:04.143721 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:04.143671 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:04.402595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:04.398961 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:04.619266 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:04.619230 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 04 01:21:04.619545 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:04.619515 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" podUID="ab9122ac-8874-4a64-9137-a7fe056db11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 04 01:21:04.688579 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:04.688463 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:21:04.726595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:04.693676 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:04.731685 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:04.731646 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:04.732282 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:04.732259 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:21:04.771703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:04.771652 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:04.811588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:04.807815 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:04.867590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:04.864404 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 01:21:04.867590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:04.864469 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" podUID="719d829c-c70a-4bdc-a3d2-5b4cb52a45da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 01:21:04.935010 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:04.934978 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 04 01:21:04.936172 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:04.936131 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" podUID="a5c642db-a1e8-4839-9fa7-1821588ecb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 04 01:21:04.996504 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:04.995721 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:04.997624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:04.997305 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:21:05.010873 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:05.010770 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:05.030635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:05.030594 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 04 01:21:05.030930 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:05.030875 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" podUID="6b03d6b4-7003-4705-b3df-64461213d3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 04 01:21:05.148774 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:05.147666 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:05.443453 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:05.443233 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:05.615591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:05.613894 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 04 01:21:05.615591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:05.613950 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" podUID="ab9122ac-8874-4a64-9137-a7fe056db11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 04 01:21:05.695670 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:05.694651 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:05.729589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:05.728659 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:05.746907 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:05.735636 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 04 01:21:05.815591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:05.815411 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:21:05.816586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:05.816317 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:21:05.873321 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:05.873276 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 01:21:05.873618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:05.873593 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" podUID="719d829c-c70a-4bdc-a3d2-5b4cb52a45da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 01:21:05.875398 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:05.874444 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:05.875398 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:05.874802 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:21:05.925901 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:05.925466 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 04 01:21:05.925901 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:05.925519 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" podUID="a5c642db-a1e8-4839-9fa7-1821588ecb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 04 01:21:06.011841 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:06.009167 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 04 01:21:06.011841 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:06.009521 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:21:06.011841 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:06.011352 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:21:06.012679 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:06.012472 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:06.051601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:06.041056 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 04 01:21:06.051601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:06.041113 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" podUID="6b03d6b4-7003-4705-b3df-64461213d3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 04 01:21:06.163400 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:06.159870 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:06.210264 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:06.209212 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:06.445747 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:06.445652 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:21:06.574406 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:06.574376 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:21:06.619535 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:06.619503 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 04 01:21:06.637130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:06.637097 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" podUID="ab9122ac-8874-4a64-9137-a7fe056db11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 04 01:21:06.637367 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:06.634256 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:21:06.704634 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:06.700163 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:06.770953 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:06.736866 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:06.770953 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:06.751771 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:21:06.770953 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:06.756842 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:21:06.883586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:06.883086 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 01:21:06.883586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:06.883141 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" podUID="719d829c-c70a-4bdc-a3d2-5b4cb52a45da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 01:21:07.020587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:07.015652 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:21:07.026866 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:07.026830 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 04 01:21:07.027044 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:07.026890 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" podUID="a5c642db-a1e8-4839-9fa7-1821588ecb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 04 01:21:07.042260 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:07.042232 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 04 01:21:07.042471 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:07.042446 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" podUID="6b03d6b4-7003-4705-b3df-64461213d3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 04 01:21:07.062686 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:07.062646 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:07.165587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:07.135242 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:07.191598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:07.188596 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:21:07.241596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:07.239900 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:21:07.340635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:07.334787 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:07.546609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:07.543707 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:07.614645 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:07.610860 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 04 01:21:07.614645 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:07.610920 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" podUID="ab9122ac-8874-4a64-9137-a7fe056db11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 04 01:21:07.704588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:07.703666 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:07.757586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:07.753659 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:07.757586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:07.754898 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:07.764594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:07.759631 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:07.853589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:07.853459 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 01:21:07.853589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:07.853512 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" podUID="719d829c-c70a-4bdc-a3d2-5b4cb52a45da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 01:21:07.947648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:07.938197 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 04 01:21:07.947648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:07.938258 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" podUID="a5c642db-a1e8-4839-9fa7-1821588ecb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 04 01:21:08.022753 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:08.020799 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:21:08.028188 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:08.028156 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 04 01:21:08.028440 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:08.028415 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" podUID="6b03d6b4-7003-4705-b3df-64461213d3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 04 01:21:08.040148 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:08.040115 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:08.200524 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:08.200426 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:21:08.562684 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:08.544717 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:08.614736 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:08.614684 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:08.628155 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:08.628122 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 04 01:21:08.656868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:08.656822 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" podUID="ab9122ac-8874-4a64-9137-a7fe056db11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 04 01:21:08.657079 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:08.655100 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:08.706792 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:08.706728 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:08.768616 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:08.767849 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:08.768616 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:08.768287 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:21:08.854227 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:08.854130 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 04 01:21:08.894995 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:08.894951 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 01:21:08.895197 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:08.895029 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" podUID="719d829c-c70a-4bdc-a3d2-5b4cb52a45da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 01:21:08.935110 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:08.935074 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 04 01:21:08.937187 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:08.937129 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" podUID="a5c642db-a1e8-4839-9fa7-1821588ecb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 04 01:21:08.974599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:08.971430 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:09.024763 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:09.024501 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:09.043592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:09.039673 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:09.080744 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:09.068224 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 04 01:21:09.081225 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:09.080997 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" podUID="6b03d6b4-7003-4705-b3df-64461213d3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 04 01:21:09.207803 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:09.207694 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:09.232224 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:09.232189 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:21:09.518205 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:09.515065 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:21:09.525900 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:09.525861 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:21:09.540010 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:09.539981 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:21:09.542535 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:09.542512 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:21:09.543990 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:09.543966 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:21:09.630763 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:09.630731 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 04 01:21:09.631162 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:09.631102 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" podUID="ab9122ac-8874-4a64-9137-a7fe056db11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 04 01:21:09.721590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:09.719248 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:09.845638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:09.844802 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 01:21:09.845638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:09.844858 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" podUID="719d829c-c70a-4bdc-a3d2-5b4cb52a45da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 01:21:09.920589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:09.919646 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:09.925611 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:09.923989 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 04 01:21:09.925611 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:09.924042 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" podUID="a5c642db-a1e8-4839-9fa7-1821588ecb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 04 01:21:10.030003 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:10.029306 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 04 01:21:10.030003 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:10.029362 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" podUID="6b03d6b4-7003-4705-b3df-64461213d3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 04 01:21:10.644590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:10.633518 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:10.921040 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:10.920944 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 04 01:21:10.921040 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:10.921004 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" podUID="a5c642db-a1e8-4839-9fa7-1821588ecb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 04 01:21:11.120058 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:11.120027 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:21:11.122417 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:11.122392 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:21:11.124107 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:11.124089 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:21:11.501587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:11.501085 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:11.624806 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:11.624075 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:11.624806 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:11.624133 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" podUID="ab9122ac-8874-4a64-9137-a7fe056db11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:11.855389 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:11.855356 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:21:11.856258 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:11.856237 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:21:11.871615 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:11.868855 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:21:11.895704 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:11.895672 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:11.895964 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:11.895938 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" podUID="719d829c-c70a-4bdc-a3d2-5b4cb52a45da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:12.071589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.070490 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:21:12.072436 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.072323 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:21:12.076726 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.076704 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:21:12.133961 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.132832 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m"] Nov 04 01:21:12.142615 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.140842 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.250622 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.240552 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m"] Nov 04 01:21:12.254676 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.254512 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:12.269069 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.269043 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:21:12.271889 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.271871 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:21:12.280842 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.280819 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:21:12.351829 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.351796 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.352020 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.351859 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.352020 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.351946 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.352142 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.352038 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.352142 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.352095 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.352342 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.352327 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.352473 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.352455 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.352528 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.352486 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.352599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.352586 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g67p\" (UniqueName: \"kubernetes.io/projected/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-kube-api-access-8g67p\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.454062 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.453986 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.454062 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.454044 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.454273 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.454064 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.454273 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.454164 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g67p\" (UniqueName: \"kubernetes.io/projected/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-kube-api-access-8g67p\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.454423 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.454276 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.454423 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.454372 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.454531 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.454498 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.454612 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.454541 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.454612 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.454578 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.454736 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.454613 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.454911 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.454888 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.454976 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.454499 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.454976 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.454920 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.454976 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.454921 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.457470 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.457440 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.457728 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.457710 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.463217 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.463196 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.463321 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.463255 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g67p\" (UniqueName: \"kubernetes.io/projected/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-kube-api-access-8g67p\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.565683 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.565649 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:12.853418 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.853390 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m"] Nov 04 01:21:12.855778 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:21:12.855752 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4cd0527_183f_484b_8dc5_8fc84aa45b1b.slice/crio-0966764d92b6e801648f8a66713ecc463f7898d9a339d9720b7aae2806a73422 WatchSource:0}: Error finding container 0966764d92b6e801648f8a66713ecc463f7898d9a339d9720b7aae2806a73422: Status 404 returned error can't find the container with id 0966764d92b6e801648f8a66713ecc463f7898d9a339d9720b7aae2806a73422 Nov 04 01:21:12.858444 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.858410 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:12.858617 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.858483 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:12.858617 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:12.858525 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:13.134683 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:13.134175 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:13.885650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:13.885603 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" event={"ID":"a4cd0527-183f-484b-8dc5-8fc84aa45b1b","Type":"ContainerStarted","Data":"ddd324c2e3ccaa3c0a89a02b2431d59a01c232f50d5e0649a1cf34ac4d2b1964"} Nov 04 01:21:13.885873 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:13.885702 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" event={"ID":"a4cd0527-183f-484b-8dc5-8fc84aa45b1b","Type":"ContainerStarted","Data":"0966764d92b6e801648f8a66713ecc463f7898d9a339d9720b7aae2806a73422"} Nov 04 01:21:13.906856 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:13.906807 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" podStartSLOduration=1.906793905 podStartE2EDuration="1.906793905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:13.905880274 +0000 UTC m=+1356.020990394" watchObservedRunningTime="2026-11-04 01:21:13.906793905 +0000 UTC m=+1356.021904023" Nov 04 01:21:13.971597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:13.971172 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:14.574589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:14.573405 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:14.574589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:14.573518 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 04 01:21:14.574589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:14.573581 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" podUID="a4cd0527-183f-484b-8dc5-8fc84aa45b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 04 01:21:14.758586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:14.754040 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:15.368210 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:15.368083 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:15.568588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:15.567977 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 04 01:21:15.568588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:15.568041 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" podUID="a4cd0527-183f-484b-8dc5-8fc84aa45b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 04 01:21:15.745640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:15.741898 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:16.574588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:16.570957 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 04 01:21:16.574588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:16.571031 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" podUID="a4cd0527-183f-484b-8dc5-8fc84aa45b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 04 01:21:16.761076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:16.761042 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:17.572024 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:17.571984 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 04 01:21:17.581821 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:17.581767 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" podUID="a4cd0527-183f-484b-8dc5-8fc84aa45b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 04 01:21:17.693228 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:17.693196 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:18.571726 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:18.571688 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 04 01:21:18.572348 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:18.572317 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" podUID="a4cd0527-183f-484b-8dc5-8fc84aa45b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 04 01:21:18.578245 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:18.578216 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:19.332696 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:19.332657 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:19.504281 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:19.503594 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:19.855278 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:19.855234 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:19.855923 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:19.855900 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:19.856807 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:19.856787 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:21:20.156632 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:20.156520 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:21.141188 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:21.141154 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:21.908731 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:21.908701 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:22.529627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:22.527333 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:22.853612 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:22.853509 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:23.320588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:23.320154 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:24.129550 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:24.129513 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:24.946510 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:24.946477 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:25.705059 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:25.705018 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:25.853017 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:25.852987 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:21:26.556748 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:26.556718 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:28.296469 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:28.296436 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:28.538735 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:28.538684 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:28.606875 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:28.606663 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:28.649468 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:28.649426 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:28.730692 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:28.730649 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:28.758692 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:28.758647 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:29.009959 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:29.005714 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:21:29.009959 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:29.006743 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:29.012712 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:29.012642 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:29.422751 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:29.422648 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:29.559513 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:29.559471 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 04 01:21:29.622381 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:29.622350 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:21:29.722908 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:29.721671 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:29.776618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:29.750602 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 04 01:21:29.776618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:29.751104 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:21:29.815591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:29.778383 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:29.815591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:29.778825 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:21:29.877930 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:29.877886 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:30.099642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:30.056583 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:30.191590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:30.191360 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:21:30.405642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:30.404996 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:30.548808 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:30.548768 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:21:30.549959 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:30.549937 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:21:30.560587 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:21:30.556088 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:21:30.569545 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:30.569515 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:30.623940 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:30.622841 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:30.758586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:30.756306 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:21:30.758586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:30.758137 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:21:30.759409 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:30.759107 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 04 01:21:30.786859 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:30.784994 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:31.058722 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:31.058621 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:31.059315 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:31.059289 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:31.059740 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:31.059720 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:21:31.158678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:31.157456 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:31.213841 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:31.213797 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:21:31.379651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:31.377102 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:21:31.379651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:31.378031 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:31.424591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:31.420477 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:21:31.435010 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:31.434036 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:21:31.453392 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:31.453361 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:21:31.462389 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:31.462258 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:21:31.534826 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:31.534789 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:32.068599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:32.068498 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:32.872754 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:32.872716 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:33.660815 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:33.660760 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:34.561208 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:34.561151 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:35.232250 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:35.232201 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:35.934591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:35.930381 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:36.544261 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.544221 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p"] Nov 04 01:21:36.549828 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.549807 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:36.559430 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.559385 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p"] Nov 04 01:21:36.559938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.559915 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d7e45d\"/\"distinct-w2-d7e45d-kserve-self-signed-certs\"" Nov 04 01:21:36.610974 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.610938 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-model-cache\") pod \"distinct-w2-d7e45d-kserve-7664dd8886-xs76p\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:36.611158 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.610986 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-dshm\") pod \"distinct-w2-d7e45d-kserve-7664dd8886-xs76p\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:36.611277 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.611247 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-tls-certs\") pod \"distinct-w2-d7e45d-kserve-7664dd8886-xs76p\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:36.611344 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.611321 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz2tl\" (UniqueName: \"kubernetes.io/projected/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-kube-api-access-fz2tl\") pod \"distinct-w2-d7e45d-kserve-7664dd8886-xs76p\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:36.611417 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.611402 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-home\") pod \"distinct-w2-d7e45d-kserve-7664dd8886-xs76p\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:36.611514 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.611498 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-tmp-dir\") pod \"distinct-w2-d7e45d-kserve-7664dd8886-xs76p\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:36.639253 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.639227 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:36.720428 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.720388 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-tmp-dir\") pod \"distinct-w2-d7e45d-kserve-7664dd8886-xs76p\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:36.720845 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.720823 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-model-cache\") pod \"distinct-w2-d7e45d-kserve-7664dd8886-xs76p\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:36.720994 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.720976 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-dshm\") pod \"distinct-w2-d7e45d-kserve-7664dd8886-xs76p\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:36.721134 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.721120 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-tls-certs\") pod \"distinct-w2-d7e45d-kserve-7664dd8886-xs76p\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:36.721243 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.721227 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz2tl\" (UniqueName: \"kubernetes.io/projected/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-kube-api-access-fz2tl\") pod \"distinct-w2-d7e45d-kserve-7664dd8886-xs76p\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:36.724690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.724662 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-home\") pod \"distinct-w2-d7e45d-kserve-7664dd8886-xs76p\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:36.730797 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.721018 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-tmp-dir\") pod \"distinct-w2-d7e45d-kserve-7664dd8886-xs76p\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:36.730986 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.721267 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-model-cache\") pod \"distinct-w2-d7e45d-kserve-7664dd8886-xs76p\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:36.752587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.725877 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-home\") pod \"distinct-w2-d7e45d-kserve-7664dd8886-xs76p\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:36.752587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.744261 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-dshm\") pod \"distinct-w2-d7e45d-kserve-7664dd8886-xs76p\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:36.780586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.767756 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz2tl\" (UniqueName: \"kubernetes.io/projected/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-kube-api-access-fz2tl\") pod \"distinct-w2-d7e45d-kserve-7664dd8886-xs76p\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:36.790621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.785247 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-tls-certs\") pod \"distinct-w2-d7e45d-kserve-7664dd8886-xs76p\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:36.806644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.799934 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:36.870302 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:36.870269 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:37.153761 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:37.153729 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p"] Nov 04 01:21:37.155066 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:21:37.155019 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3becce48_d40e_4ee9_9a2f_b553bec9f2c5.slice/crio-437952ee9c8d56bf4b2dbb938f203cab9dcef3d7f8e7552842f629e56dfefd55 WatchSource:0}: Error finding container 437952ee9c8d56bf4b2dbb938f203cab9dcef3d7f8e7552842f629e56dfefd55: Status 404 returned error can't find the container with id 437952ee9c8d56bf4b2dbb938f203cab9dcef3d7f8e7552842f629e56dfefd55 Nov 04 01:21:37.257700 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:37.257659 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" event={"ID":"3becce48-d40e-4ee9-9a2f-b553bec9f2c5","Type":"ContainerStarted","Data":"a84d99d9111bb846c2b4fb4770ed555ea310ff064d855527fe0a749c07c05863"} Nov 04 01:21:37.257861 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:37.257716 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" event={"ID":"3becce48-d40e-4ee9-9a2f-b553bec9f2c5","Type":"ContainerStarted","Data":"437952ee9c8d56bf4b2dbb938f203cab9dcef3d7f8e7552842f629e56dfefd55"} Nov 04 01:21:37.274877 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:37.274830 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podStartSLOduration=1.274812325 podStartE2EDuration="1.274812325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:37.27367946 +0000 UTC m=+1379.388789578" watchObservedRunningTime="2026-11-04 01:21:37.274812325 +0000 UTC m=+1379.389922444" Nov 04 01:21:37.411903 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:37.411828 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:37.871305 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:37.871279 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:38.547496 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:38.545742 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:39.242590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:39.241142 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:39.953587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:39.949143 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:40.751745 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:40.740176 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:41.494775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:41.493294 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:42.313283 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.313240 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:42.476468 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.476433 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr"] Nov 04 01:21:42.480808 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.480792 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.492527 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.492504 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr"] Nov 04 01:21:42.514513 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.514480 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb"] Nov 04 01:21:42.519351 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.519306 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.534727 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.533492 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb"] Nov 04 01:21:42.635999 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.635923 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.636136 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.636006 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.636136 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.636050 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.636136 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.636094 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.636275 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.636226 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.636275 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.636250 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.636275 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.636267 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.636379 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.636346 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0958a62-b79b-4d8f-9a10-81acf1522245-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.636415 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.636381 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhc9p\" (UniqueName: \"kubernetes.io/projected/c0958a62-b79b-4d8f-9a10-81acf1522245-kube-api-access-dhc9p\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.636415 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.636398 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvj2q\" (UniqueName: \"kubernetes.io/projected/93308cb8-8e7f-45c3-a1f6-bbd045723e67-kube-api-access-mvj2q\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.636485 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.636419 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.636525 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.636490 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.636525 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.636521 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.636639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.636625 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.636677 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.636661 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.636803 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.636773 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.636937 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.636824 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.636937 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.636871 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.677074 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.677050 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf"] Nov 04 01:21:42.681034 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.681017 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.693991 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.693968 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf"] Nov 04 01:21:42.737740 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.737705 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.737740 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.737745 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.738076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.737764 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.738076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.737791 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.738076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.737858 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.738076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.737883 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.738076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.737911 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.738076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.737966 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.738076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.737996 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.738076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.738020 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.738076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.738051 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0958a62-b79b-4d8f-9a10-81acf1522245-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.738604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.738105 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhc9p\" (UniqueName: \"kubernetes.io/projected/c0958a62-b79b-4d8f-9a10-81acf1522245-kube-api-access-dhc9p\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.738604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.738128 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvj2q\" (UniqueName: \"kubernetes.io/projected/93308cb8-8e7f-45c3-a1f6-bbd045723e67-kube-api-access-mvj2q\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.738604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.738161 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.738604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.738207 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.738604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.738239 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.738604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.738343 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.738604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.738408 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.738981 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.738609 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.738981 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.738657 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.738981 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.738869 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.738981 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.738915 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.739400 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.738996 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.739400 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.739167 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.739400 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.739273 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.739621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.739494 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0958a62-b79b-4d8f-9a10-81acf1522245-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.739621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.739499 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.740932 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.740906 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.741351 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.741322 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.741557 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.741531 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.741773 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.741742 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.741867 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.741795 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.747418 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.747391 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.747754 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.747733 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvj2q\" (UniqueName: \"kubernetes.io/projected/93308cb8-8e7f-45c3-a1f6-bbd045723e67-kube-api-access-mvj2q\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.747975 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.747960 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.748196 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.748180 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhc9p\" (UniqueName: \"kubernetes.io/projected/c0958a62-b79b-4d8f-9a10-81acf1522245-kube-api-access-dhc9p\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.795472 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.795451 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:42.840061 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.840034 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.840198 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.840078 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.840198 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.840098 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.840198 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.840163 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.840198 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.840181 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.840333 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.840201 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.840333 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.840253 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.840333 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.840282 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.840333 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.840316 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf25q\" (UniqueName: \"kubernetes.io/projected/cae606b8-4a95-4046-a9eb-3df149dcf7a8-kube-api-access-pf25q\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.843618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.843599 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:42.941048 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.941020 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.941191 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.941091 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.941191 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.941122 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.941191 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.941154 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf25q\" (UniqueName: \"kubernetes.io/projected/cae606b8-4a95-4046-a9eb-3df149dcf7a8-kube-api-access-pf25q\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.941368 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.941230 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.941368 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.941281 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.941368 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.941308 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.941524 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.941415 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.941524 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.941447 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.941524 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.941491 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.941705 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.941539 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.942371 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.941822 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.942371 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.942068 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.942371 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.942147 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.945194 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.945172 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.945330 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.945315 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.950162 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.950131 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.950252 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.950233 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf25q\" (UniqueName: \"kubernetes.io/projected/cae606b8-4a95-4046-a9eb-3df149dcf7a8-kube-api-access-pf25q\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:42.996406 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:42.996375 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:43.080690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:43.080663 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr"] Nov 04 01:21:43.083455 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:21:43.082908 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93308cb8_8e7f_45c3_a1f6_bbd045723e67.slice/crio-083dd58c53950cd264db79f91ed6348c436d8843c6c9da1a7aba84bc1e8e09c2 WatchSource:0}: Error finding container 083dd58c53950cd264db79f91ed6348c436d8843c6c9da1a7aba84bc1e8e09c2: Status 404 returned error can't find the container with id 083dd58c53950cd264db79f91ed6348c436d8843c6c9da1a7aba84bc1e8e09c2 Nov 04 01:21:43.085721 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:43.085684 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:43.085840 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:43.085764 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:43.085840 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:43.085797 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:43.160533 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:43.160339 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:43.208818 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:43.207140 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:43.208818 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:43.207227 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:43.208818 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:43.207269 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:43.208818 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:43.208272 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb"] Nov 04 01:21:43.290131 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:21:43.290096 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae606b8_4a95_4046_a9eb_3df149dcf7a8.slice/crio-31090203615dd32de09b4623a0cbfab2f65bc0a41f4816548b25cda7719c6432 WatchSource:0}: Error finding container 31090203615dd32de09b4623a0cbfab2f65bc0a41f4816548b25cda7719c6432: Status 404 returned error can't find the container with id 31090203615dd32de09b4623a0cbfab2f65bc0a41f4816548b25cda7719c6432 Nov 04 01:21:43.290720 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:43.290689 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf"] Nov 04 01:21:43.293383 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:43.293341 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:43.293462 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:43.293446 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:43.293514 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:43.293499 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:43.330932 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:43.330898 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" event={"ID":"93308cb8-8e7f-45c3-a1f6-bbd045723e67","Type":"ContainerStarted","Data":"66314a9ef0a948f5d32fb90a73843dda7ca8b2eca8ae141e77129c52f4c5be76"} Nov 04 01:21:43.332498 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:43.332476 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" event={"ID":"93308cb8-8e7f-45c3-a1f6-bbd045723e67","Type":"ContainerStarted","Data":"083dd58c53950cd264db79f91ed6348c436d8843c6c9da1a7aba84bc1e8e09c2"} Nov 04 01:21:43.334872 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:43.334853 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" event={"ID":"cae606b8-4a95-4046-a9eb-3df149dcf7a8","Type":"ContainerStarted","Data":"31090203615dd32de09b4623a0cbfab2f65bc0a41f4816548b25cda7719c6432"} Nov 04 01:21:43.341122 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:43.340490 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" event={"ID":"c0958a62-b79b-4d8f-9a10-81acf1522245","Type":"ContainerStarted","Data":"be1b10acb88d6fe373bfa9a321293554c5d732bed733b08d4d26cd9ffc3e5a93"} Nov 04 01:21:43.341122 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:43.340539 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" event={"ID":"c0958a62-b79b-4d8f-9a10-81acf1522245","Type":"ContainerStarted","Data":"d0c30ed8b7dc5991dede8152d98d98e43727ec37a270aefb776b647f5ae75fe0"} Nov 04 01:21:43.359736 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:43.359678 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" podStartSLOduration=1.359662185 podStartE2EDuration="1.359662185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:43.354921608 +0000 UTC m=+1385.470031727" watchObservedRunningTime="2026-11-04 01:21:43.359662185 +0000 UTC m=+1385.474772388" Nov 04 01:21:43.393932 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:43.393875 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" podStartSLOduration=1.393860469 podStartE2EDuration="1.393860469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:43.386688312 +0000 UTC m=+1385.501798431" watchObservedRunningTime="2026-11-04 01:21:43.393860469 +0000 UTC m=+1385.508970587" Nov 04 01:21:43.796508 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:43.796475 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:43.844369 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:43.844349 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:44.119209 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:44.119120 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:44.397746 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:44.397655 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" event={"ID":"cae606b8-4a95-4046-a9eb-3df149dcf7a8","Type":"ContainerStarted","Data":"94b72b3e9f3931e8c1101a8f0969b818516cefe4ad899d9a7d6d2f46d3e78203"} Nov 04 01:21:44.433837 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:44.423162 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" podStartSLOduration=2.423144975 podStartE2EDuration="2.423144975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:44.422447777 +0000 UTC m=+1386.537557895" watchObservedRunningTime="2026-11-04 01:21:44.423144975 +0000 UTC m=+1386.538255094" Nov 04 01:21:44.810804 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:44.810765 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 04 01:21:44.811118 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:44.811088 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" podUID="93308cb8-8e7f-45c3-a1f6-bbd045723e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 04 01:21:44.845717 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:44.845678 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 04 01:21:44.845927 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:44.845738 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" podUID="c0958a62-b79b-4d8f-9a10-81acf1522245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 04 01:21:44.997662 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:44.997629 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:44.998528 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:44.998494 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:21:44.998685 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:44.998552 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" podUID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 04 01:21:45.109653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:45.109499 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:45.796409 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:45.795949 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 04 01:21:45.796409 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:45.796014 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" podUID="93308cb8-8e7f-45c3-a1f6-bbd045723e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 04 01:21:45.847599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:45.845054 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 04 01:21:45.847599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:45.845107 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" podUID="c0958a62-b79b-4d8f-9a10-81acf1522245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 04 01:21:45.997463 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:45.997004 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:21:45.997463 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:45.997065 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" podUID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 04 01:21:46.001202 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:46.001059 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:46.798042 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:46.798007 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 04 01:21:46.798647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:46.798619 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" podUID="93308cb8-8e7f-45c3-a1f6-bbd045723e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 04 01:21:46.843974 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:46.843939 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 04 01:21:46.844229 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:46.844205 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" podUID="c0958a62-b79b-4d8f-9a10-81acf1522245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 04 01:21:46.873589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:46.872696 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:46.909586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:46.909111 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:46.998982 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:46.998946 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:21:46.999326 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:46.999286 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" podUID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 04 01:21:47.455799 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:47.455756 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:47.556132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:47.556080 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:47.698254 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:47.698209 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:47.832664 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:47.832605 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 04 01:21:47.858606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:47.852232 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 04 01:21:47.858606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:47.852290 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" podUID="c0958a62-b79b-4d8f-9a10-81acf1522245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 04 01:21:47.858985 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:47.858937 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" podUID="93308cb8-8e7f-45c3-a1f6-bbd045723e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 04 01:21:48.047589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:48.023032 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:21:48.052690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:48.052642 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" podUID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 04 01:21:48.245474 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:48.245187 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:48.795936 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:48.795893 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 04 01:21:48.796139 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:48.795962 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" podUID="93308cb8-8e7f-45c3-a1f6-bbd045723e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 04 01:21:48.844512 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:48.844388 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 04 01:21:48.844512 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:48.844438 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" podUID="c0958a62-b79b-4d8f-9a10-81acf1522245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 04 01:21:48.999589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:48.999038 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:21:48.999589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:48.999098 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" podUID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 04 01:21:49.191937 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:49.191812 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:49.797590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:49.796644 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 04 01:21:49.797590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:49.796703 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" podUID="93308cb8-8e7f-45c3-a1f6-bbd045723e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 04 01:21:49.843920 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:49.843888 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 04 01:21:49.844092 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:49.843941 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" podUID="c0958a62-b79b-4d8f-9a10-81acf1522245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 04 01:21:50.006919 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:50.006883 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:21:50.007469 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:50.007439 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" podUID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 04 01:21:50.150302 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:50.150197 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:50.797967 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:50.797932 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 04 01:21:50.798238 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:50.798211 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" podUID="93308cb8-8e7f-45c3-a1f6-bbd045723e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 04 01:21:50.854589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:50.849945 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 04 01:21:50.854589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:50.850013 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" podUID="c0958a62-b79b-4d8f-9a10-81acf1522245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 04 01:21:51.002597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:50.998248 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:21:51.002597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:50.998298 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" podUID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 04 01:21:51.559651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:51.559480 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:51.807590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:51.797929 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 04 01:21:51.807590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:51.797993 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" podUID="93308cb8-8e7f-45c3-a1f6-bbd045723e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 04 01:21:51.848645 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:51.847051 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 04 01:21:51.848645 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:51.847112 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" podUID="c0958a62-b79b-4d8f-9a10-81acf1522245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 04 01:21:52.009590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:52.008878 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:21:52.009590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:52.008950 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" podUID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 04 01:21:52.390589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:52.390190 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:52.390820 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:52.390713 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:52.432589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:52.431295 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:52.476588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:52.476385 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:52.723645 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:52.720717 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:52.723645 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:52.721886 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:21:52.763601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:52.760872 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 04 01:21:52.796726 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:52.794131 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:52.802585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:52.798660 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:52.802585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:52.799170 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 04 01:21:52.802585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:52.799207 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" podUID="93308cb8-8e7f-45c3-a1f6-bbd045723e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 04 01:21:52.850588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:52.844111 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 04 01:21:52.850588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:52.844189 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" podUID="c0958a62-b79b-4d8f-9a10-81acf1522245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 04 01:21:53.022683 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:53.021209 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:53.036950 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:53.036910 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 04 01:21:53.090588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:53.083627 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:53.152703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:53.152657 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:53.459644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:53.458811 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:21:53.463589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:53.459793 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:21:53.758644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:53.745497 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 04 01:21:53.758644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:53.745894 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:21:53.758644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:53.746475 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:21:53.758644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:53.746528 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:21:53.810590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:53.809672 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 04 01:21:53.845589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:53.845437 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:21:54.062665 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:53.918788 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 04 01:21:54.062665 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:53.919230 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:21:54.062665 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:54.049647 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 04 01:21:54.079609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:54.070926 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:21:54.097626 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:54.086388 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.097626 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:54.095037 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:54.097626 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:54.095101 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" podUID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.097626 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:54.095671 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.213590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:54.207341 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:21:54.213590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:54.207403 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:21:54.213590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:54.207439 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:21:54.384641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:54.378974 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:54.458763 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:54.458723 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.484736 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:54.466642 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.485226 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:54.484753 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 04 01:21:54.718526 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:54.718436 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:54.749336 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:54.748665 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.780947 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:54.753647 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 04 01:21:54.813918 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:54.813873 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 04 01:21:54.894408 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:54.894369 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:54.920732 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:54.920692 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 04 01:21:54.936062 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:54.934940 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:21:54.988115 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:54.988029 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:54.988398 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:54.988365 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" podUID="c0958a62-b79b-4d8f-9a10-81acf1522245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.022729 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:55.022101 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:55.022729 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:55.022180 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" podUID="93308cb8-8e7f-45c3-a1f6-bbd045723e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.051023 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:55.049804 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.076606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:55.071644 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.146765 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:55.146729 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:55.162761 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:55.162705 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" podUID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.216876 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:55.216832 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.489729 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:55.489682 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 04 01:21:55.490401 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:55.489980 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:21:55.491850 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:55.490029 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.499681 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:55.491034 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:55.503686 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:55.503662 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:21:55.785649 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:55.768855 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 04 01:21:55.785649 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:55.776692 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:21:56.036643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.035823 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 04 01:21:56.059595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.055670 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.059595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.056411 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:56.059595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.056453 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" podUID="93308cb8-8e7f-45c3-a1f6-bbd045723e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.101595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.088739 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:56.101595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.088805 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" podUID="c0958a62-b79b-4d8f-9a10-81acf1522245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.101595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.096959 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:21:56.152167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.152114 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:21:56.205773 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.205721 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:56.225705 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.206028 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" podUID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.228723 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.228684 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.257693 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.257090 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:56.477121 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.477038 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:21:56.495963 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.495923 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.502034 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.501992 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:21:56.502674 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.502643 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 04 01:21:56.630590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.579522 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:56.630590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.614627 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:56.693461 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.693431 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:21:56.724901 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.724866 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:21:56.779103 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.777391 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.779103 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.777804 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 04 01:21:56.856618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.833007 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.858833 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.858806 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:56.884589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.870965 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:56.884589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.871063 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:21:56.912959 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:56.912926 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:21:57.077628 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.043469 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:57.078035 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.062235 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.098726 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.098656 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.250667 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.245383 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:57.250667 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.245453 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" podUID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.410480 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.408992 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": context deadline exceeded" Nov 04 01:21:57.507703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.507653 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:21:57.517685 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.517648 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.636988 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.632040 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:21:57.688694 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.688612 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c"] Nov 04 01:21:57.699188 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.693480 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c"] Nov 04 01:21:57.699188 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.693552 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.699188 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.693922 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:57.725227 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.713449 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:21:57.749621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.746987 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:21:57.805771 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.805542 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:21:57.806495 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.806472 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:21:57.864118 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.864075 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.876042 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.875996 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:21:57.902652 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.897400 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:21:57.907166 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.907144 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:21:57.909134 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.909112 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2lv7\" (UniqueName: \"kubernetes.io/projected/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-kube-api-access-b2lv7\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:57.909340 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.909318 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:57.932924 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.915498 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:57.933798 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.933771 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:57.934337 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.934317 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:57.934536 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.934505 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:57.940974 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.935542 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:57.941149 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.941132 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:57.941583 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.941541 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:57.959605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:57.954019 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:21:58.052438 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.052352 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:58.091638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.076925 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:58.091638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.076992 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:58.091638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.077050 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:58.091638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.077159 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2lv7\" (UniqueName: \"kubernetes.io/projected/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-kube-api-access-b2lv7\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:58.091638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.077188 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:58.091638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.077286 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:58.091638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.077422 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:58.091638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.077456 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:58.091638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.077512 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:58.091638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.078278 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:58.091638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.078822 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:58.091638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.079016 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:58.091638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.079333 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:58.091638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.084084 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:58.104378 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.104157 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2lv7\" (UniqueName: \"kubernetes.io/projected/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-kube-api-access-b2lv7\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:58.104531 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.104414 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:58.109925 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.109895 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:58.122593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.120986 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:58.274585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.268371 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:58.288711 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.288675 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:58.289630 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.288974 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" podUID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:58.359264 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.359227 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:58.366075 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.366048 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:21:58.492140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.491864 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:21:58.667305 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.667229 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c"] Nov 04 01:21:58.670414 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:21:58.670374 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda247fcaf_f6ae_499c_83d9_acfdfa09cad0.slice/crio-324f02e82bcc9e2a6a1214aac718de2e3fd141f0c60e73d70e959c778b7fdedb WatchSource:0}: Error finding container 324f02e82bcc9e2a6a1214aac718de2e3fd141f0c60e73d70e959c778b7fdedb: Status 404 returned error can't find the container with id 324f02e82bcc9e2a6a1214aac718de2e3fd141f0c60e73d70e959c778b7fdedb Nov 04 01:21:58.673233 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.673182 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:58.673403 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.673272 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:58.673403 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:58.673311 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:59.080027 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:59.079993 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:21:59.499397 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:59.499309 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" event={"ID":"a247fcaf-f6ae-499c-83d9-acfdfa09cad0","Type":"ContainerStarted","Data":"7bc34e24d741f58a32e121838c46b49a793759d33225049adffbea76091b4ee2"} Nov 04 01:21:59.499397 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:59.499372 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" event={"ID":"a247fcaf-f6ae-499c-83d9-acfdfa09cad0","Type":"ContainerStarted","Data":"324f02e82bcc9e2a6a1214aac718de2e3fd141f0c60e73d70e959c778b7fdedb"} Nov 04 01:21:59.524511 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:21:59.524458 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" podStartSLOduration=2.524441694 podStartE2EDuration="2.524441694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:59.522261594 +0000 UTC m=+1401.637371713" watchObservedRunningTime="2026-11-04 01:21:59.524441694 +0000 UTC m=+1401.639551813" Nov 04 01:21:59.887803 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:21:59.887765 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:22:00.052595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:00.051821 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:00.372161 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:00.372128 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:22:00.372854 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:00.372829 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 04 01:22:00.373022 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:00.372996 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" podUID="a247fcaf-f6ae-499c-83d9-acfdfa09cad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 04 01:22:00.936692 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:00.936655 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:01.371952 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:01.371914 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 04 01:22:01.372216 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:01.372186 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" podUID="a247fcaf-f6ae-499c-83d9-acfdfa09cad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 04 01:22:01.845599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:01.844666 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:02.373594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:02.371891 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 04 01:22:02.373594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:02.371948 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" podUID="a247fcaf-f6ae-499c-83d9-acfdfa09cad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 04 01:22:02.763590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:02.762786 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:03.376590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:03.375915 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 04 01:22:03.376590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:03.375975 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" podUID="a247fcaf-f6ae-499c-83d9-acfdfa09cad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 04 01:22:03.653609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:03.653509 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:04.369592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:04.367921 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 04 01:22:04.369592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:04.367979 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" podUID="a247fcaf-f6ae-499c-83d9-acfdfa09cad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 04 01:22:04.639650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:04.638120 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:05.367760 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:05.367725 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 04 01:22:05.367959 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:05.367786 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" podUID="a247fcaf-f6ae-499c-83d9-acfdfa09cad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 04 01:22:05.421455 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:05.421421 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:05.605588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:05.604587 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:06.441286 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:06.441254 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:06.604126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:06.604100 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:22:06.604741 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:06.604672 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:22:06.605694 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:06.605674 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:22:07.210104 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:07.210066 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:07.357591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:07.352065 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:08.259588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:08.257503 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:08.925452 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:08.925419 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:09.829595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:09.828978 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:10.827386 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:10.813290 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:11.656594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:11.654798 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:12.445136 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.445091 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:12.663150 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.663120 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d"] Nov 04 01:22:12.669197 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.669175 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.680022 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.679993 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d"] Nov 04 01:22:12.839438 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.839407 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbc69a29-d23c-4da4-a6af-be04977ee457-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.839622 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.839457 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.839670 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.839633 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.839670 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.839665 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s79x4\" (UniqueName: \"kubernetes.io/projected/fbc69a29-d23c-4da4-a6af-be04977ee457-kube-api-access-s79x4\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.839739 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.839687 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.839784 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.839770 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.839818 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.839803 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.839851 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.839837 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.839940 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.839928 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.941261 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.941232 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.941448 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.941356 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.941448 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.941394 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s79x4\" (UniqueName: \"kubernetes.io/projected/fbc69a29-d23c-4da4-a6af-be04977ee457-kube-api-access-s79x4\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.941448 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.941419 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.941650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.941549 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.941650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.941598 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.941650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.941629 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.941786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.941730 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.941786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.941741 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.941786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.941744 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.941919 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.941831 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbc69a29-d23c-4da4-a6af-be04977ee457-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.942115 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.942093 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.942169 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.942095 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.942532 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.942508 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbc69a29-d23c-4da4-a6af-be04977ee457-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.944098 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.944077 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.944297 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.944278 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.949861 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.949842 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.950078 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.950061 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s79x4\" (UniqueName: \"kubernetes.io/projected/fbc69a29-d23c-4da4-a6af-be04977ee457-kube-api-access-s79x4\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:12.993108 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:12.993080 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:13.119897 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:13.119814 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:13.285374 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:13.285346 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d"] Nov 04 01:22:13.287064 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:22:13.287036 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc69a29_d23c_4da4_a6af_be04977ee457.slice/crio-5540814cf0403bf2a06acfe1eed949afc965567043e2a451c1fbe5f6f1edd0aa WatchSource:0}: Error finding container 5540814cf0403bf2a06acfe1eed949afc965567043e2a451c1fbe5f6f1edd0aa: Status 404 returned error can't find the container with id 5540814cf0403bf2a06acfe1eed949afc965567043e2a451c1fbe5f6f1edd0aa Nov 04 01:22:13.290093 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:13.290002 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:13.290197 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:13.290102 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:13.290197 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:13.290146 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:13.622987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:13.622948 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" event={"ID":"fbc69a29-d23c-4da4-a6af-be04977ee457","Type":"ContainerStarted","Data":"3fd939b9ff86004e1acc4c88457314d2ca6faa02cbc7e28c2b59eacfe8bb2613"} Nov 04 01:22:13.623188 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:13.623016 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" event={"ID":"fbc69a29-d23c-4da4-a6af-be04977ee457","Type":"ContainerStarted","Data":"5540814cf0403bf2a06acfe1eed949afc965567043e2a451c1fbe5f6f1edd0aa"} Nov 04 01:22:13.650260 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:13.650208 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" podStartSLOduration=1.650194352 podStartE2EDuration="1.650194352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:13.648299209 +0000 UTC m=+1415.763409329" watchObservedRunningTime="2026-11-04 01:22:13.650194352 +0000 UTC m=+1415.765304470" Nov 04 01:22:13.997224 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:13.997146 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:14.009032 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:14.009000 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:14.994606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:14.994554 4852 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 04 01:22:14.994786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:14.994630 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" podUID="fbc69a29-d23c-4da4-a6af-be04977ee457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 04 01:22:15.049591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:15.044031 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:15.777536 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:15.777437 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:15.994300 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:15.994267 4852 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 04 01:22:15.994489 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:15.994338 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" podUID="fbc69a29-d23c-4da4-a6af-be04977ee457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 04 01:22:16.572585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:16.565115 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:16.993902 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:16.993825 4852 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 04 01:22:16.993902 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:16.993882 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" podUID="fbc69a29-d23c-4da4-a6af-be04977ee457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 04 01:22:17.386587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:17.382777 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:17.993399 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:17.993367 4852 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 04 01:22:17.993897 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:17.993411 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" podUID="fbc69a29-d23c-4da4-a6af-be04977ee457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 04 01:22:18.130958 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:18.130924 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:18.993955 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:18.993921 4852 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 04 01:22:18.994349 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:18.993972 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" podUID="fbc69a29-d23c-4da4-a6af-be04977ee457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 04 01:22:19.228076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:19.228041 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:20.348026 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:20.348000 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:20.349338 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:20.349307 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:20.350196 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:20.350175 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:22:20.507587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:20.506389 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:21.329037 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:21.326415 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:22.145350 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:22.145316 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:22.835172 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:22.835137 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:23.296711 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:23.296684 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:22:23.628183 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:23.628090 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:24.529587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:24.528963 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:25.510598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:25.509817 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:25.699591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:25.697672 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:25.776674 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:25.776542 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:26.780725 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:26.780675 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:26.918743 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:26.918700 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:26.924682 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:26.924646 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 04 01:22:26.962716 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:26.962667 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.081043 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:27.080939 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.145713 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:27.145671 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:22:27.355544 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:27.355448 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.356588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:27.356532 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.556604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:27.494660 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:22:27.556604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:27.503418 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.556604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:27.516204 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.701672 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:27.697945 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.771781 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:27.771719 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.802621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:27.793235 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.802621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:27.793758 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:22:27.895723 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:27.895678 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Nov 04 01:22:27.921065 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:27.921020 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.929591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:27.921738 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:22:27.979524 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:27.979415 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.987404 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:27.987374 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:22:28.106593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:28.089727 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:28.202025 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:28.162361 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:28.349903 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:28.349864 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:28.414737 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:28.414690 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" podUID="719d829c-c70a-4bdc-a3d2-5b4cb52a45da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:28.505702 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:28.503683 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:28.528781 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:28.528736 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 04 01:22:28.656804 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:28.656701 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:28.737672 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:28.699334 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:22:28.737672 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:28.699392 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:22:28.737672 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:28.699427 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:22:28.737672 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:28.699466 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:22:28.765664 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:28.765075 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:28.773594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:28.766244 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:22:28.778591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:28.773876 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:28.795594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:28.787050 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:22:28.931651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:28.902888 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:28.944604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:28.933175 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:28.990590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:28.950728 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 04 01:22:29.088603 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:28.503977 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:29.111616 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:29.101668 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:29.111616 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:29.103418 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:29.172795 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:29.172684 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:29.435368 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:29.435337 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:22:29.541625 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:29.534255 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:29.628591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:29.544677 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:22:29.862594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:29.788140 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:29.909591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:29.822193 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:30.004617 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:29.959787 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": context deadline exceeded" Nov 04 01:22:30.004617 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:29.960345 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:22:30.004617 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:29.968864 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:22:30.437588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:30.433500 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:30.466809 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:30.438509 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:22:30.468030 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:30.460006 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:30.468533 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:30.460045 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:22:30.469739 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:30.460069 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:30.470321 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:30.460088 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 04 01:22:30.470699 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:30.460106 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:22:30.557860 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:30.557817 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:30.682060 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:22:30.682018 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:22:30.705829 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:30.705707 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded" Nov 04 01:22:30.715685 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:30.715649 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:22:30.723988 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:30.723964 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:22:31.124591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:31.097283 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:22:31.173530 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:31.173481 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 04 01:22:31.380432 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:31.373794 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:22:31.385119 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:31.385098 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:22:31.390493 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:31.390420 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:22:31.401765 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:31.399356 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:31.408091 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:31.408057 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:22:31.408894 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:31.408868 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:22:31.420285 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:31.420264 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:22:31.487740 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:31.487706 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:22:31.488114 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:31.488067 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:31.834976 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:31.834911 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:22:31.929591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:31.926358 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:22:31.952617 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:31.950832 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:31.952617 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:31.951793 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:22:31.974335 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:31.973539 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:22:32.146228 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:32.146134 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:32.755224 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:32.755188 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:33.440188 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:33.440154 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:34.228324 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:34.228292 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:35.250157 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:35.250122 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:36.021121 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:36.021045 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:36.743606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:36.737066 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:37.522591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:37.515538 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:38.452365 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:38.452066 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:39.236288 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:39.234454 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:40.051746 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:40.051704 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:40.782689 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:40.782622 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:41.679248 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:41.679190 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:42.526339 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.526300 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:42.767931 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.767896 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6"] Nov 04 01:22:42.774676 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.774634 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.797543 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.797481 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6"] Nov 04 01:22:42.821739 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.821711 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp"] Nov 04 01:22:42.826971 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.826949 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:42.830846 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.830815 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.830976 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.830871 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrmhw\" (UniqueName: \"kubernetes.io/projected/baf9a879-211d-43e5-82ba-3909c2ee1b33-kube-api-access-hrmhw\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.830976 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.830908 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.830976 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.830966 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.831165 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.831022 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.831225 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.831204 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.831281 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.831233 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.831336 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.831289 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf9a879-211d-43e5-82ba-3909c2ee1b33-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.831390 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.831332 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.857631 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.857601 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp"] Nov 04 01:22:42.897667 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.897635 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f"] Nov 04 01:22:42.902509 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.902489 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:42.923346 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.923321 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f"] Nov 04 01:22:42.932482 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.932452 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.932652 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.932508 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrmhw\" (UniqueName: \"kubernetes.io/projected/baf9a879-211d-43e5-82ba-3909c2ee1b33-kube-api-access-hrmhw\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.932652 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.932543 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.932652 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.932620 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:42.932848 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.932671 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:42.932848 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.932706 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.932848 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.932750 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:42.932848 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.932793 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.932848 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.932831 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx6p8\" (UniqueName: \"kubernetes.io/projected/ad7e7c06-6bfc-4fee-a35a-43b73e501458-kube-api-access-wx6p8\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:42.933107 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.932854 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.933107 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.932874 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:42.933107 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.932900 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:42.933107 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.933100 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:42.933307 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.933148 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.933307 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.933167 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.933307 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.933202 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf9a879-211d-43e5-82ba-3909c2ee1b33-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.933307 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.933230 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.933307 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.933261 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:42.933307 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.933281 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:42.933635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.933525 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.934390 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.933764 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.934390 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.933800 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.934390 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.934350 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf9a879-211d-43e5-82ba-3909c2ee1b33-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.935514 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.935484 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.936904 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.936875 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.944948 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.944924 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:42.945219 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:42.945198 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrmhw\" (UniqueName: \"kubernetes.io/projected/baf9a879-211d-43e5-82ba-3909c2ee1b33-kube-api-access-hrmhw\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:43.034825 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.034751 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.034963 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.034887 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.034963 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.034912 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df08c3cb-cf91-4bec-a39d-c32e07739abf-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.034963 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.034939 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:43.034963 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.034959 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:43.035174 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.035068 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:43.035174 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.035103 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.035174 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.035163 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:43.035337 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.035189 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.035337 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.035274 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.035449 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.035338 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:43.035449 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.035388 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bkhm\" (UniqueName: \"kubernetes.io/projected/df08c3cb-cf91-4bec-a39d-c32e07739abf-kube-api-access-4bkhm\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.035449 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.035405 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:43.035658 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.035481 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx6p8\" (UniqueName: \"kubernetes.io/projected/ad7e7c06-6bfc-4fee-a35a-43b73e501458-kube-api-access-wx6p8\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:43.035658 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.035527 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:43.035658 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.035558 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:43.035658 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.035649 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.035928 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.035732 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:43.035928 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.035855 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:43.035928 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.035880 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.036078 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.035946 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:43.036078 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.035970 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:43.036278 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.036258 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:43.037541 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.037518 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:43.037541 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.037529 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:43.043860 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.043841 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx6p8\" (UniqueName: \"kubernetes.io/projected/ad7e7c06-6bfc-4fee-a35a-43b73e501458-kube-api-access-wx6p8\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:43.044083 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.044063 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:43.091409 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.091384 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:43.137352 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.137320 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.137527 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.137393 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.137527 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.137476 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.137670 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.137539 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.137670 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.137582 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df08c3cb-cf91-4bec-a39d-c32e07739abf-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.137670 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.137665 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.137827 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.137713 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.137827 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.137744 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.137827 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.137752 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.137827 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.137795 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bkhm\" (UniqueName: \"kubernetes.io/projected/df08c3cb-cf91-4bec-a39d-c32e07739abf-kube-api-access-4bkhm\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.138021 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.137868 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.138085 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.138019 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.138299 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.138199 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.138299 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.138249 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df08c3cb-cf91-4bec-a39d-c32e07739abf-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.141854 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.141831 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.141948 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.141922 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.149235 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.146647 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bkhm\" (UniqueName: \"kubernetes.io/projected/df08c3cb-cf91-4bec-a39d-c32e07739abf-kube-api-access-4bkhm\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.149457 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.149432 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.166650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.166616 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:43.225579 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.225520 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:43.389528 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:22:43.389494 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf9a879_211d_43e5_82ba_3909c2ee1b33.slice/crio-9bba0632d5739a86f6de6b261cfab07bbb41789f411eba0740e9f9ca49814477 WatchSource:0}: Error finding container 9bba0632d5739a86f6de6b261cfab07bbb41789f411eba0740e9f9ca49814477: Status 404 returned error can't find the container with id 9bba0632d5739a86f6de6b261cfab07bbb41789f411eba0740e9f9ca49814477 Nov 04 01:22:43.389964 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.389940 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6"] Nov 04 01:22:43.392228 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.392198 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:43.392286 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.392264 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:43.392326 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.392292 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:43.466634 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.463430 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:43.540383 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.540360 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp"] Nov 04 01:22:43.541183 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:22:43.541160 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7e7c06_6bfc_4fee_a35a_43b73e501458.slice/crio-e8929607325bf07e0f390a17029713285d711e5905f266761f77b542986d2fd3 WatchSource:0}: Error finding container e8929607325bf07e0f390a17029713285d711e5905f266761f77b542986d2fd3: Status 404 returned error can't find the container with id e8929607325bf07e0f390a17029713285d711e5905f266761f77b542986d2fd3 Nov 04 01:22:43.544624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.544544 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:43.544703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.544654 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:43.544703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.544685 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:43.563118 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.563089 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f"] Nov 04 01:22:43.564861 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:22:43.564836 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf08c3cb_cf91_4bec_a39d_c32e07739abf.slice/crio-3b33be0c98fc29c4a216f6fefccb1cfa621ff9b07672b114c46d7e063ef8a05f WatchSource:0}: Error finding container 3b33be0c98fc29c4a216f6fefccb1cfa621ff9b07672b114c46d7e063ef8a05f: Status 404 returned error can't find the container with id 3b33be0c98fc29c4a216f6fefccb1cfa621ff9b07672b114c46d7e063ef8a05f Nov 04 01:22:43.567116 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.567086 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:43.567191 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.567173 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:43.567235 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.567201 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:43.626794 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.626711 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" event={"ID":"df08c3cb-cf91-4bec-a39d-c32e07739abf","Type":"ContainerStarted","Data":"3b33be0c98fc29c4a216f6fefccb1cfa621ff9b07672b114c46d7e063ef8a05f"} Nov 04 01:22:43.628746 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.628641 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" event={"ID":"ad7e7c06-6bfc-4fee-a35a-43b73e501458","Type":"ContainerStarted","Data":"e8929607325bf07e0f390a17029713285d711e5905f266761f77b542986d2fd3"} Nov 04 01:22:43.634112 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.634092 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" event={"ID":"baf9a879-211d-43e5-82ba-3909c2ee1b33","Type":"ContainerStarted","Data":"4ebd7897ca3f7299cf4740281616c06cb505ba9ab0442e8a1db672a9e98ced27"} Nov 04 01:22:43.634226 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.634136 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" event={"ID":"baf9a879-211d-43e5-82ba-3909c2ee1b33","Type":"ContainerStarted","Data":"9bba0632d5739a86f6de6b261cfab07bbb41789f411eba0740e9f9ca49814477"} Nov 04 01:22:43.656512 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:43.656277 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" podStartSLOduration=1.656258319 podStartE2EDuration="1.656258319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:43.654741637 +0000 UTC m=+1445.769851755" watchObservedRunningTime="2026-11-04 01:22:43.656258319 +0000 UTC m=+1445.771368438" Nov 04 01:22:44.092219 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:44.092190 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:44.093422 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:44.093399 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:22:44.093525 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:44.093454 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" podUID="baf9a879-211d-43e5-82ba-3909c2ee1b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 04 01:22:44.352670 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:44.352583 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:44.653862 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:44.653768 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" event={"ID":"df08c3cb-cf91-4bec-a39d-c32e07739abf","Type":"ContainerStarted","Data":"2bc3fdbad492290277c5ab697666237aba16f9910ad6ae894f80497a5fa6615c"} Nov 04 01:22:44.655531 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:44.655503 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" event={"ID":"ad7e7c06-6bfc-4fee-a35a-43b73e501458","Type":"ContainerStarted","Data":"a8194384632e59c0650c08d6e51a91c98c567f33611eb537e3874f6bc5edba97"} Nov 04 01:22:44.712590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:44.711957 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" podStartSLOduration=2.711939067 podStartE2EDuration="2.711939067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:44.700056391 +0000 UTC m=+1446.815166510" watchObservedRunningTime="2026-11-04 01:22:44.711939067 +0000 UTC m=+1446.827049186" Nov 04 01:22:44.750707 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:44.750653 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" podStartSLOduration=2.750636719 podStartE2EDuration="2.750636719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:44.747480736 +0000 UTC m=+1446.862590854" watchObservedRunningTime="2026-11-04 01:22:44.750636719 +0000 UTC m=+1446.865746836" Nov 04 01:22:45.104591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:45.097473 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:22:45.104591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:45.097536 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" podUID="baf9a879-211d-43e5-82ba-3909c2ee1b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 04 01:22:45.177591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:45.169778 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:45.177591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:45.175675 4852 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:22:45.177591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:45.175731 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" podUID="ad7e7c06-6bfc-4fee-a35a-43b73e501458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 04 01:22:45.235168 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:45.235027 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:45.236441 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:45.236412 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:22:45.236679 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:45.236653 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" podUID="df08c3cb-cf91-4bec-a39d-c32e07739abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 01:22:45.633591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:45.631115 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:46.134182 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:46.134143 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:22:46.134802 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:46.134767 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" podUID="baf9a879-211d-43e5-82ba-3909c2ee1b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 04 01:22:46.224416 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:46.224382 4852 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:22:46.224936 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:46.224904 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" podUID="ad7e7c06-6bfc-4fee-a35a-43b73e501458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 04 01:22:46.228928 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:46.228904 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:22:46.229080 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:46.229058 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" podUID="df08c3cb-cf91-4bec-a39d-c32e07739abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 01:22:47.100000 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:47.099962 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:22:47.100249 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:47.100223 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" podUID="baf9a879-211d-43e5-82ba-3909c2ee1b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 04 01:22:47.143588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:47.143184 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:47.252271 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:47.184641 4852 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:22:47.252271 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:47.184698 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" podUID="ad7e7c06-6bfc-4fee-a35a-43b73e501458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 04 01:22:47.307693 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:47.285715 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:22:47.307693 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:47.285779 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" podUID="df08c3cb-cf91-4bec-a39d-c32e07739abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 01:22:48.130073 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:48.130040 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:22:48.143714 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:48.143646 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" podUID="baf9a879-211d-43e5-82ba-3909c2ee1b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 04 01:22:48.178378 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:48.178345 4852 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:22:48.207874 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:48.207836 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" podUID="ad7e7c06-6bfc-4fee-a35a-43b73e501458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 04 01:22:48.403773 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:48.403675 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:22:48.421604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:48.403985 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" podUID="df08c3cb-cf91-4bec-a39d-c32e07739abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 01:22:49.108589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:49.107853 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:22:49.108589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:49.107912 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" podUID="baf9a879-211d-43e5-82ba-3909c2ee1b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 04 01:22:49.178593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:49.175937 4852 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:22:49.178593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:49.175994 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" podUID="ad7e7c06-6bfc-4fee-a35a-43b73e501458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 04 01:22:49.233721 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:49.233683 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:22:49.234017 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:49.233985 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" podUID="df08c3cb-cf91-4bec-a39d-c32e07739abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 01:22:49.424642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:49.422440 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:50.092396 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:50.091943 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:22:50.092396 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:50.092005 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" podUID="baf9a879-211d-43e5-82ba-3909c2ee1b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 04 01:22:50.173589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:50.172899 4852 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:22:50.173589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:50.172959 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" podUID="ad7e7c06-6bfc-4fee-a35a-43b73e501458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 04 01:22:50.236132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:50.236093 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:22:50.236846 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:50.236811 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" podUID="df08c3cb-cf91-4bec-a39d-c32e07739abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 01:22:51.091879 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:51.091842 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:22:51.092066 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:51.091902 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" podUID="baf9a879-211d-43e5-82ba-3909c2ee1b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 04 01:22:51.167704 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:51.167670 4852 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:22:51.167963 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:51.167933 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" podUID="ad7e7c06-6bfc-4fee-a35a-43b73e501458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 04 01:22:51.227047 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:51.227013 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:22:51.227199 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:51.227079 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" podUID="df08c3cb-cf91-4bec-a39d-c32e07739abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 01:22:52.068003 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:52.067962 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:52.112070 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:52.111907 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:52.114147 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:52.114125 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:52.117142 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:52.117090 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:22:52.178253 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:52.178220 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:52.179836 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:52.179810 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:52.184021 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:52.183998 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:22:52.232439 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:52.232409 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:52.233274 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:52.233249 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:52.234911 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:52.234585 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:22:53.244795 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:53.244763 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:54.258309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:54.258275 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:55.253680 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:55.253627 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:56.027601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:56.025239 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:57.056255 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:57.056180 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:22:58.399591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:58.395958 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:58.607191 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:58.606691 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:58.632590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:58.628690 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:58.922810 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:58.922764 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.030591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:59.014649 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.113256 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:59.106037 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:22:59.113256 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:59.108934 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.160590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:59.159729 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.398610 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:59.397667 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.398610 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:59.398300 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:22:59.422592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:59.420898 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.473590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:59.472650 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.497588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:59.496892 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.570585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:59.569669 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.692649 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:59.692462 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:22:59.717590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:59.692901 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:22:59.806591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:59.806052 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:59.806591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:59.806110 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" podUID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.887592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:59.883762 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.969641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:59.951919 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.969641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:22:59.952882 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:23:00.044591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:00.043791 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:23:00.135592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:00.134799 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:00.135592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:00.135274 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:23:00.186741 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:00.186687 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:00.359529 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:00.359486 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": context deadline exceeded" Nov 04 01:23:00.456874 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:00.456827 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:00.522372 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:00.491225 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:00.522372 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:00.502843 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:00.522372 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:00.503802 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 04 01:23:00.578351 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:00.578272 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:00.602952 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:00.602885 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 04 01:23:00.791206 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:00.791164 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:23:00.978080 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:00.978034 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:23:01.010322 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:23:01.010287 4852 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.023s" Nov 04 01:23:01.010796 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:01.010776 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:23:01.018909 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:01.018886 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:23:01.019109 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:01.019094 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:23:01.019257 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:01.019244 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:23:01.019379 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:01.019366 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:23:01.019520 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:01.019508 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:23:01.028075 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:01.027650 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:23:01.028075 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:01.027716 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:23:01.072075 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:01.063317 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:01.148542 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:01.143240 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 04 01:23:01.148542 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:01.147646 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:23:01.232590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:01.231957 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:01.313591 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:23:01.283739 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:23:01.534698 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:01.472715 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:01.568591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:01.498750 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:01.590589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:01.503864 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:01.591185 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:01.504995 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:01.614588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:01.603646 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 04 01:23:01.793802 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:01.793697 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:01.913011 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:01.912654 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:02.028120 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:02.028074 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:23:02.144825 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:02.144710 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:02.145462 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:02.145431 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:02.147923 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:02.147896 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:23:02.235792 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:02.234766 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:02.449651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:02.442666 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:02.544685 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:02.543664 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:02.569742 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:02.569694 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:02.596321 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:02.594664 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:02.596321 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:02.595017 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:23:02.604677 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:02.604640 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:02.798769 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:02.797669 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": context deadline exceeded" Nov 04 01:23:02.798769 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:02.797780 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:23:03.030590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:03.028653 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:03.149651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:03.145661 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:03.149651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:03.146726 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:03.174721 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:03.174666 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:03.238719 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:03.238672 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:03.494659 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:03.493851 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:23:03.558818 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:03.555938 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:03.608711 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:03.608659 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:03.626733 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:03.626683 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 04 01:23:03.642599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:03.627326 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 04 01:23:03.651404 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:03.629929 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:23:03.807749 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:03.807697 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:03.979618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:03.979583 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:04.019090 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:04.019048 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:04.035938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:04.035893 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:04.150896 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:04.150788 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:04.178756 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:04.178706 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:04.255937 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:04.255841 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 04 01:23:04.559725 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:04.559668 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:04.614950 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:04.614897 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:04.628712 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:04.628661 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:04.645691 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:04.645645 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:04.651980 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:04.651946 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:04.812811 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:04.812689 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:23:05.040106 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:05.039119 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:23:05.155978 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:05.155867 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.156827 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:05.156793 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.181713 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:05.181656 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.258866 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:05.258820 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 04 01:23:05.567586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:05.565785 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.615803 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:05.615739 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.644767 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:05.631547 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.646292 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:05.646244 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.656128 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:05.655675 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.814732 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:05.814685 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:06.052590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:06.046674 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:23:06.165722 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:06.165673 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:06.166578 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:06.166529 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:23:06.168449 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:06.166791 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:06.183708 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:06.183667 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:06.302033 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:06.301993 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 04 01:23:06.591590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:06.586678 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:06.591590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:06.586747 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" podUID="baefa3c0-28be-428a-9591-f0ca1f07fafa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:06.592160 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:06.591650 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:23:06.631899 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:06.631858 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:06.647341 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:06.647265 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:06.675628 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:06.656755 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:06.675628 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:06.657107 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:06.823721 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:06.823668 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:06.899544 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:06.899448 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:23:07.049780 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:07.049720 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:07.178658 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:07.172647 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:07.247587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:07.200120 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:07.465331 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:07.462506 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:07.485899 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:07.482939 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:07.494259 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:07.494201 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": context deadline exceeded" Nov 04 01:23:07.593647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:07.592710 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:23:07.639613 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:07.636745 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 04 01:23:07.648362 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:07.647985 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 04 01:23:07.703852 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:07.702662 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 04 01:23:07.703852 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:07.702985 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:07.830946 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:07.830917 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:23:07.831401 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:07.831383 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:23:07.831584 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:07.831551 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:23:07.863420 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:07.863394 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:23:07.864679 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:07.864646 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:23:07.868907 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:07.868890 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:23:07.902581 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:07.896796 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:23:07.905413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:07.905374 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:07.908521 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:07.908500 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:23:07.971857 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:07.971053 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:23:08.018383 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:08.018312 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:08.036951 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:08.036919 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:23:09.163690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:09.163660 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:09.971048 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:09.971015 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:10.650591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:10.650147 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:11.370587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:11.364806 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:12.088591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:12.083207 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:12.896590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:12.892155 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:14.632433 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:14.632397 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:14.854684 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:14.854651 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:15.193253 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:15.193218 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:15.970605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:15.970477 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:16.840254 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:16.840219 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:17.669653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:17.669618 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:18.358047 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:18.358009 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:19.041315 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:19.041219 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:19.739206 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:19.736553 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:20.630424 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:20.630385 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:21.365588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:21.364364 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:22.066973 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:22.066906 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:22.842674 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:22.842553 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:23.622100 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:23.622044 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:24.457285 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:24.455220 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:25.159589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:25.155724 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:25.460583 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:25.460488 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:23:25.859585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:25.855236 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:26.658121 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:26.658086 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:27.594600 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:27.583173 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:29.057268 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:29.024341 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:29.117613 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:29.098882 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:29.197627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:29.187673 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:23:29.263726 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:29.260660 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:29.575336 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:29.575260 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:23:29.590114 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:29.587654 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:29.601987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:29.601953 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:29.602237 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:29.602206 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" podUID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:29.638597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:29.637719 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:29.638597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:29.638355 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:29.655772 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:29.655740 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:23:29.658513 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:29.658486 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 04 01:23:29.831807 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:29.831708 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:23:29.887278 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:29.885337 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:29.983018 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:23:29.982981 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:23:30.011903 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:30.009636 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:30.306905 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:30.306867 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:31.054056 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:31.054018 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:31.851174 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:31.851133 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:32.271136 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:32.271108 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:32.647671 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:32.647534 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:33.462595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:33.450230 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:34.160600 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:34.160469 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:34.962142 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:34.962105 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:35.669734 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:35.669688 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:36.082978 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:36.082936 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq"] Nov 04 01:23:36.087128 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:36.087107 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq" Nov 04 01:23:36.092925 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:36.092905 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:23:36.093254 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:36.093152 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:23:36.098451 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:36.098077 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq"] Nov 04 01:23:36.212713 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:36.212685 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/863dc71f-0017-4800-81a0-053404d652d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b61ce18-429mq\" (UID: \"863dc71f-0017-4800-81a0-053404d652d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq" Nov 04 01:23:36.212958 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:36.212848 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/863dc71f-0017-4800-81a0-053404d652d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b61ce18-429mq\" (UID: \"863dc71f-0017-4800-81a0-053404d652d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq" Nov 04 01:23:36.213035 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:36.212963 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wrj5\" (UniqueName: \"kubernetes.io/projected/863dc71f-0017-4800-81a0-053404d652d4-kube-api-access-6wrj5\") pod \"maas-api-revoke-sub-1b61ce18-429mq\" (UID: \"863dc71f-0017-4800-81a0-053404d652d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq" Nov 04 01:23:36.313847 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:36.313818 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/863dc71f-0017-4800-81a0-053404d652d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b61ce18-429mq\" (UID: \"863dc71f-0017-4800-81a0-053404d652d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq" Nov 04 01:23:36.314030 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:36.313937 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/863dc71f-0017-4800-81a0-053404d652d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b61ce18-429mq\" (UID: \"863dc71f-0017-4800-81a0-053404d652d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq" Nov 04 01:23:36.314030 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:36.313998 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wrj5\" (UniqueName: \"kubernetes.io/projected/863dc71f-0017-4800-81a0-053404d652d4-kube-api-access-6wrj5\") pod \"maas-api-revoke-sub-1b61ce18-429mq\" (UID: \"863dc71f-0017-4800-81a0-053404d652d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq" Nov 04 01:23:36.314598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:36.314546 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/863dc71f-0017-4800-81a0-053404d652d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b61ce18-429mq\" (UID: \"863dc71f-0017-4800-81a0-053404d652d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq" Nov 04 01:23:36.316437 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:36.316416 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/863dc71f-0017-4800-81a0-053404d652d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b61ce18-429mq\" (UID: \"863dc71f-0017-4800-81a0-053404d652d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq" Nov 04 01:23:36.325602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:36.325579 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wrj5\" (UniqueName: \"kubernetes.io/projected/863dc71f-0017-4800-81a0-053404d652d4-kube-api-access-6wrj5\") pod \"maas-api-revoke-sub-1b61ce18-429mq\" (UID: \"863dc71f-0017-4800-81a0-053404d652d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq" Nov 04 01:23:36.406235 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:36.406158 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq" Nov 04 01:23:36.686006 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:36.685980 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq"] Nov 04 01:23:36.686970 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:23:36.686943 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863dc71f_0017_4800_81a0_053404d652d4.slice/crio-e1c60827123da19d612b5dc9cf73aa0bae0dc38fc160d47dcfe38089ebb834f6 WatchSource:0}: Error finding container e1c60827123da19d612b5dc9cf73aa0bae0dc38fc160d47dcfe38089ebb834f6: Status 404 returned error can't find the container with id e1c60827123da19d612b5dc9cf73aa0bae0dc38fc160d47dcfe38089ebb834f6 Nov 04 01:23:37.000736 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:37.000706 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq" event={"ID":"863dc71f-0017-4800-81a0-053404d652d4","Type":"ContainerStarted","Data":"baf96c31a392bdddcfab9cbbf1d1523c3bbedf9466a6f46beb26a5400d2319ee"} Nov 04 01:23:37.000903 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:37.000770 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq" event={"ID":"863dc71f-0017-4800-81a0-053404d652d4","Type":"ContainerStarted","Data":"e1c60827123da19d612b5dc9cf73aa0bae0dc38fc160d47dcfe38089ebb834f6"} Nov 04 01:23:37.020466 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:37.020421 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq" podStartSLOduration=1.020407885 podStartE2EDuration="1.020407885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:37.017889086 +0000 UTC m=+1499.132999206" watchObservedRunningTime="2026-11-04 01:23:37.020407885 +0000 UTC m=+1499.135518047" Nov 04 01:23:38.009688 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:38.009657 4852 generic.go:356] "Generic (PLEG): container finished" podID="863dc71f-0017-4800-81a0-053404d652d4" containerID="baf96c31a392bdddcfab9cbbf1d1523c3bbedf9466a6f46beb26a5400d2319ee" exitCode=0 Nov 04 01:23:38.010084 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:38.009736 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq" event={"ID":"863dc71f-0017-4800-81a0-053404d652d4","Type":"ContainerDied","Data":"baf96c31a392bdddcfab9cbbf1d1523c3bbedf9466a6f46beb26a5400d2319ee"} Nov 04 01:23:38.185193 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:38.185167 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 01:23:38.187702 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:38.187683 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 01:23:38.430657 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:38.430558 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:38.730640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:38.730350 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:39.189507 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:39.189483 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq" Nov 04 01:23:39.276574 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:39.276480 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/863dc71f-0017-4800-81a0-053404d652d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/863dc71f-0017-4800-81a0-053404d652d4-maas-api-xks-ca\") pod \"863dc71f-0017-4800-81a0-053404d652d4\" (UID: \"863dc71f-0017-4800-81a0-053404d652d4\") " Nov 04 01:23:39.276750 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:39.276690 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/863dc71f-0017-4800-81a0-053404d652d4-kube-api-access-6wrj5\" (UniqueName: \"kubernetes.io/projected/863dc71f-0017-4800-81a0-053404d652d4-kube-api-access-6wrj5\") pod \"863dc71f-0017-4800-81a0-053404d652d4\" (UID: \"863dc71f-0017-4800-81a0-053404d652d4\") " Nov 04 01:23:39.276823 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:39.276813 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/863dc71f-0017-4800-81a0-053404d652d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/863dc71f-0017-4800-81a0-053404d652d4-maas-api-openshift-ca\") pod \"863dc71f-0017-4800-81a0-053404d652d4\" (UID: \"863dc71f-0017-4800-81a0-053404d652d4\") " Nov 04 01:23:39.277176 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:39.277146 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/863dc71f-0017-4800-81a0-053404d652d4-maas-api-openshift-ca" pod "863dc71f-0017-4800-81a0-053404d652d4" (UID: "863dc71f-0017-4800-81a0-053404d652d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:23:39.277317 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:39.277303 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/863dc71f-0017-4800-81a0-053404d652d4-maas-api-openshift-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:39.278644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:39.278615 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/863dc71f-0017-4800-81a0-053404d652d4-maas-api-xks-ca" pod "863dc71f-0017-4800-81a0-053404d652d4" (UID: "863dc71f-0017-4800-81a0-053404d652d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:23:39.279191 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:39.279164 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863dc71f-0017-4800-81a0-053404d652d4-kube-api-access-6wrj5" pod "863dc71f-0017-4800-81a0-053404d652d4" (UID: "863dc71f-0017-4800-81a0-053404d652d4"). InnerVolumeSpecName "kube-api-access-6wrj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:23:39.378996 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:39.378967 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/863dc71f-0017-4800-81a0-053404d652d4-maas-api-xks-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:39.378996 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:39.378995 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wrj5\" (UniqueName: \"kubernetes.io/projected/863dc71f-0017-4800-81a0-053404d652d4-kube-api-access-6wrj5\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:39.515072 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:39.515027 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:40.023412 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:40.023385 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq" Nov 04 01:23:40.023576 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:40.023414 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq" event={"ID":"863dc71f-0017-4800-81a0-053404d652d4","Type":"ContainerDied","Data":"e1c60827123da19d612b5dc9cf73aa0bae0dc38fc160d47dcfe38089ebb834f6"} Nov 04 01:23:40.023576 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:40.023461 4852 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1c60827123da19d612b5dc9cf73aa0bae0dc38fc160d47dcfe38089ebb834f6" Nov 04 01:23:40.326903 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:40.326869 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:41.282807 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:41.282769 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:42.110603 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:42.110553 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:42.943654 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:42.941968 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:43.656246 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:43.656205 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:44.376678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:44.376643 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:45.127694 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:45.127656 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:45.928482 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:45.928438 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:46.815951 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:46.815919 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:47.897586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:47.892284 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:48.362752 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:48.362713 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bea7a66-xhdkc"] Nov 04 01:23:48.363335 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:48.363317 4852 cpu_manager.go:414] "containerMap: removing container" podUID="863dc71f-0017-4800-81a0-053404d652d4" containerName="revoke-keys" Nov 04 01:23:48.363335 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:48.363335 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="863dc71f-0017-4800-81a0-053404d652d4" containerName="revoke-keys" Nov 04 01:23:48.363548 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:48.363427 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="863dc71f-0017-4800-81a0-053404d652d4" containerName="revoke-keys" Nov 04 01:23:48.367653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:48.367634 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bea7a66-xhdkc" Nov 04 01:23:48.380738 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:48.377748 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:23:48.380738 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:48.377835 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:23:48.382838 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:48.382817 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bea7a66-xhdkc"] Nov 04 01:23:48.501474 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:48.501429 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxkhl\" (UniqueName: \"kubernetes.io/projected/f7609e7b-4627-4850-a0b0-0e72aa4ba39f-kube-api-access-jxkhl\") pod \"maas-api-revoke-sub-8bea7a66-xhdkc\" (UID: \"f7609e7b-4627-4850-a0b0-0e72aa4ba39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bea7a66-xhdkc" Nov 04 01:23:48.501701 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:48.501602 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7609e7b-4627-4850-a0b0-0e72aa4ba39f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bea7a66-xhdkc\" (UID: \"f7609e7b-4627-4850-a0b0-0e72aa4ba39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bea7a66-xhdkc" Nov 04 01:23:48.501701 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:48.501656 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7609e7b-4627-4850-a0b0-0e72aa4ba39f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bea7a66-xhdkc\" (UID: \"f7609e7b-4627-4850-a0b0-0e72aa4ba39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bea7a66-xhdkc" Nov 04 01:23:48.603289 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:48.603258 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxkhl\" (UniqueName: \"kubernetes.io/projected/f7609e7b-4627-4850-a0b0-0e72aa4ba39f-kube-api-access-jxkhl\") pod \"maas-api-revoke-sub-8bea7a66-xhdkc\" (UID: \"f7609e7b-4627-4850-a0b0-0e72aa4ba39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bea7a66-xhdkc" Nov 04 01:23:48.603446 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:48.603327 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7609e7b-4627-4850-a0b0-0e72aa4ba39f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bea7a66-xhdkc\" (UID: \"f7609e7b-4627-4850-a0b0-0e72aa4ba39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bea7a66-xhdkc" Nov 04 01:23:48.603446 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:48.603366 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7609e7b-4627-4850-a0b0-0e72aa4ba39f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bea7a66-xhdkc\" (UID: \"f7609e7b-4627-4850-a0b0-0e72aa4ba39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bea7a66-xhdkc" Nov 04 01:23:48.604083 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:48.604058 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7609e7b-4627-4850-a0b0-0e72aa4ba39f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bea7a66-xhdkc\" (UID: \"f7609e7b-4627-4850-a0b0-0e72aa4ba39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bea7a66-xhdkc" Nov 04 01:23:48.605801 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:48.605778 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7609e7b-4627-4850-a0b0-0e72aa4ba39f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bea7a66-xhdkc\" (UID: \"f7609e7b-4627-4850-a0b0-0e72aa4ba39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bea7a66-xhdkc" Nov 04 01:23:48.615360 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:48.615234 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxkhl\" (UniqueName: \"kubernetes.io/projected/f7609e7b-4627-4850-a0b0-0e72aa4ba39f-kube-api-access-jxkhl\") pod \"maas-api-revoke-sub-8bea7a66-xhdkc\" (UID: \"f7609e7b-4627-4850-a0b0-0e72aa4ba39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bea7a66-xhdkc" Nov 04 01:23:48.689784 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:48.689755 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bea7a66-xhdkc" Nov 04 01:23:48.746760 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:48.746660 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:49.065678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:49.065653 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bea7a66-xhdkc"] Nov 04 01:23:49.066719 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:23:49.066692 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7609e7b_4627_4850_a0b0_0e72aa4ba39f.slice/crio-01e154df4d7d02cec7e05006993fc24dcc0a27b6c4e8b14f97dba0ca013ed75c WatchSource:0}: Error finding container 01e154df4d7d02cec7e05006993fc24dcc0a27b6c4e8b14f97dba0ca013ed75c: Status 404 returned error can't find the container with id 01e154df4d7d02cec7e05006993fc24dcc0a27b6c4e8b14f97dba0ca013ed75c Nov 04 01:23:49.169051 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:49.169018 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bea7a66-xhdkc" event={"ID":"f7609e7b-4627-4850-a0b0-0e72aa4ba39f","Type":"ContainerStarted","Data":"01e154df4d7d02cec7e05006993fc24dcc0a27b6c4e8b14f97dba0ca013ed75c"} Nov 04 01:23:49.453233 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:49.453197 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:50.177067 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:50.177032 4852 generic.go:356] "Generic (PLEG): container finished" podID="f7609e7b-4627-4850-a0b0-0e72aa4ba39f" containerID="2c1cf64259645efa8913d307a0260514357b787c4e9ca38da0a6e87e0ecefec4" exitCode=0 Nov 04 01:23:50.177436 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:50.177114 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bea7a66-xhdkc" event={"ID":"f7609e7b-4627-4850-a0b0-0e72aa4ba39f","Type":"ContainerDied","Data":"2c1cf64259645efa8913d307a0260514357b787c4e9ca38da0a6e87e0ecefec4"} Nov 04 01:23:50.378099 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:50.378062 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:51.384589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:51.357915 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:51.426548 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:51.426524 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bea7a66-xhdkc" Nov 04 01:23:51.590852 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:51.590826 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7609e7b-4627-4850-a0b0-0e72aa4ba39f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7609e7b-4627-4850-a0b0-0e72aa4ba39f-maas-api-openshift-ca\") pod \"f7609e7b-4627-4850-a0b0-0e72aa4ba39f\" (UID: \"f7609e7b-4627-4850-a0b0-0e72aa4ba39f\") " Nov 04 01:23:51.591005 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:51.590878 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7609e7b-4627-4850-a0b0-0e72aa4ba39f-kube-api-access-jxkhl\" (UniqueName: \"kubernetes.io/projected/f7609e7b-4627-4850-a0b0-0e72aa4ba39f-kube-api-access-jxkhl\") pod \"f7609e7b-4627-4850-a0b0-0e72aa4ba39f\" (UID: \"f7609e7b-4627-4850-a0b0-0e72aa4ba39f\") " Nov 04 01:23:51.591162 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:51.591143 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7609e7b-4627-4850-a0b0-0e72aa4ba39f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7609e7b-4627-4850-a0b0-0e72aa4ba39f-maas-api-xks-ca\") pod \"f7609e7b-4627-4850-a0b0-0e72aa4ba39f\" (UID: \"f7609e7b-4627-4850-a0b0-0e72aa4ba39f\") " Nov 04 01:23:51.591199 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:51.591164 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7609e7b-4627-4850-a0b0-0e72aa4ba39f-maas-api-openshift-ca" pod "f7609e7b-4627-4850-a0b0-0e72aa4ba39f" (UID: "f7609e7b-4627-4850-a0b0-0e72aa4ba39f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:23:51.591723 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:51.591700 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7609e7b-4627-4850-a0b0-0e72aa4ba39f-maas-api-openshift-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:51.593220 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:51.593190 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7609e7b-4627-4850-a0b0-0e72aa4ba39f-maas-api-xks-ca" pod "f7609e7b-4627-4850-a0b0-0e72aa4ba39f" (UID: "f7609e7b-4627-4850-a0b0-0e72aa4ba39f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:23:51.593324 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:51.593226 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7609e7b-4627-4850-a0b0-0e72aa4ba39f-kube-api-access-jxkhl" pod "f7609e7b-4627-4850-a0b0-0e72aa4ba39f" (UID: "f7609e7b-4627-4850-a0b0-0e72aa4ba39f"). InnerVolumeSpecName "kube-api-access-jxkhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:23:51.692881 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:51.692857 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7609e7b-4627-4850-a0b0-0e72aa4ba39f-maas-api-xks-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:51.692881 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:51.692881 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxkhl\" (UniqueName: \"kubernetes.io/projected/f7609e7b-4627-4850-a0b0-0e72aa4ba39f-kube-api-access-jxkhl\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:52.143045 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:52.131806 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:52.209718 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:52.205696 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bea7a66-xhdkc" event={"ID":"f7609e7b-4627-4850-a0b0-0e72aa4ba39f","Type":"ContainerDied","Data":"01e154df4d7d02cec7e05006993fc24dcc0a27b6c4e8b14f97dba0ca013ed75c"} Nov 04 01:23:52.209718 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:52.205759 4852 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01e154df4d7d02cec7e05006993fc24dcc0a27b6c4e8b14f97dba0ca013ed75c" Nov 04 01:23:52.209718 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:52.205825 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bea7a66-xhdkc" Nov 04 01:23:52.829740 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:52.829705 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:53.547592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:53.544935 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:54.262684 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:54.260089 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:55.147883 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:55.147850 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:56.110804 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:56.110772 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:56.938664 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:56.938616 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:57.843664 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:57.843627 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:59.084589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:59.083164 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:59.858627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:23:59.855262 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:23:59.924607 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:23:59.924545 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:24:00.846587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:00.837767 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:01.672327 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:01.672299 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:02.393549 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:02.393508 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:03.152210 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:03.152079 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:04.080515 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:04.080465 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:04.757587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:04.750667 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:04.868040 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:04.868010 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11017-7c57c6"] Nov 04 01:24:04.868554 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:04.868542 4852 cpu_manager.go:414] "containerMap: removing container" podUID="f7609e7b-4627-4850-a0b0-0e72aa4ba39f" containerName="revoke-keys" Nov 04 01:24:04.868634 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:04.868556 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7609e7b-4627-4850-a0b0-0e72aa4ba39f" containerName="revoke-keys" Nov 04 01:24:04.868675 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:04.868661 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7609e7b-4627-4850-a0b0-0e72aa4ba39f" containerName="revoke-keys" Nov 04 01:24:04.872167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:04.872151 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11017-7c57c6" Nov 04 01:24:04.877062 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:04.877037 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-rc7dk\"" Nov 04 01:24:04.879703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:04.879681 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11017-7c57c6"] Nov 04 01:24:04.938263 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:04.938225 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7rt4\" (UniqueName: \"kubernetes.io/projected/8819a827-3fc1-44d0-93db-03287c4ee9ee-kube-api-access-r7rt4\") pod \"test-curl-11017-7c57c6\" (UID: \"8819a827-3fc1-44d0-93db-03287c4ee9ee\") " pod="opendatahub/test-curl-11017-7c57c6" Nov 04 01:24:05.039779 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:05.039706 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7rt4\" (UniqueName: \"kubernetes.io/projected/8819a827-3fc1-44d0-93db-03287c4ee9ee-kube-api-access-r7rt4\") pod \"test-curl-11017-7c57c6\" (UID: \"8819a827-3fc1-44d0-93db-03287c4ee9ee\") " pod="opendatahub/test-curl-11017-7c57c6" Nov 04 01:24:05.049024 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:05.048997 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7rt4\" (UniqueName: \"kubernetes.io/projected/8819a827-3fc1-44d0-93db-03287c4ee9ee-kube-api-access-r7rt4\") pod \"test-curl-11017-7c57c6\" (UID: \"8819a827-3fc1-44d0-93db-03287c4ee9ee\") " pod="opendatahub/test-curl-11017-7c57c6" Nov 04 01:24:05.186139 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:05.186115 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11017-7c57c6" Nov 04 01:24:05.464686 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:05.464662 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11017-7c57c6"] Nov 04 01:24:05.465437 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:24:05.465410 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8819a827_3fc1_44d0_93db_03287c4ee9ee.slice/crio-2aea61bf39840887677b02022df4751bd41cfacdcb46f0acc925f6d39ebd8e1e WatchSource:0}: Error finding container 2aea61bf39840887677b02022df4751bd41cfacdcb46f0acc925f6d39ebd8e1e: Status 404 returned error can't find the container with id 2aea61bf39840887677b02022df4751bd41cfacdcb46f0acc925f6d39ebd8e1e Nov 04 01:24:05.542634 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:05.542305 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:06.339791 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:06.339760 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11017-7c57c6" event={"ID":"8819a827-3fc1-44d0-93db-03287c4ee9ee","Type":"ContainerStarted","Data":"2aea61bf39840887677b02022df4751bd41cfacdcb46f0acc925f6d39ebd8e1e"} Nov 04 01:24:06.410741 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:06.410713 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:06.748305 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:06.748168 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:24:07.383753 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:07.383682 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:09.258285 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:09.257046 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:24:09.278602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:09.268248 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 04 01:24:09.278602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:09.268536 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:24:09.585640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:09.579661 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:09.585640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:09.580697 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:09.622590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:09.621716 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:09.666589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:09.665653 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:09.688826 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:09.680665 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:24:09.803695 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:09.803646 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:09.885241 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:09.884938 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:09.928674 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:09.927646 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:10.017590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:10.005750 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:10.288836 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:10.288774 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:10.375634 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:10.298996 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:10.375634 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:10.296094 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 04 01:24:10.375634 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:10.300719 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:10.733641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:10.567624 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:10.733641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:10.579892 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:10.733641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:10.649074 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:10.733641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:10.690957 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 04 01:24:10.792592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:10.765439 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:10.792592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:10.766971 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:24:10.811592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:10.806138 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:24:11.130620 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.120091 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:11.187595 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:24:11.177446 4852 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.025s" Nov 04 01:24:11.187595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.177672 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:24:11.187595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.177740 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:24:11.187595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.177797 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:24:11.187595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.177862 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:24:11.187595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.177924 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:24:11.187595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.177980 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:24:11.187595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.178033 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:24:11.187595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.178089 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:24:11.187595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.178140 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:24:11.187595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.178194 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:24:11.197792 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.197749 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:11.243592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.239778 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:24:11.306707 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.306662 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:24:11.308818 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.308783 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:11.354381 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.308999 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:24:11.384865 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.336709 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:24:11.823661 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.681858 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:24:11.823661 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.682262 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 04 01:24:11.823661 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.716109 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:11.844589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.825499 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:11.861878 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.840647 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:11.927671 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:11.924666 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:12.121884 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:12.121774 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:12.218596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:12.208743 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 04 01:24:12.307623 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:12.282669 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:12.307623 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:12.282734 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" podUID="6b03d6b4-7003-4705-b3df-64461213d3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:12.348592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:12.344649 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:12.435679 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:12.416046 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:12.435679 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:12.416312 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:24:12.435679 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:12.416683 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:24:12.480592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:12.465340 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:12.689833 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:12.689736 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:12.691197 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:12.689736 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:12.736993 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:12.719702 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:12.763728 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:12.763690 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:12.763992 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:12.763962 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" podUID="93308cb8-8e7f-45c3-a1f6-bbd045723e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:12.880591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:12.855626 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": context deadline exceeded" Nov 04 01:24:12.993680 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:12.881470 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:13.006705 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:12.881516 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:13.135592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:13.135531 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:13.135785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:13.135617 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" podUID="ab9122ac-8874-4a64-9137-a7fe056db11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:13.135976 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:13.135944 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:24:13.146716 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:13.146679 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:13.146970 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:13.146941 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" podUID="c0958a62-b79b-4d8f-9a10-81acf1522245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:13.215916 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:13.215865 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:13.397001 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:13.396952 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:13.431590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:13.420461 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:13.431590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:13.420836 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:24:13.431590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:13.421388 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:24:13.511589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:13.507864 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:13.700054 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:13.699950 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:13.738921 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:13.738868 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:13.801590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:13.797690 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:14.038825 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:14.030899 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:24:14.051708 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:14.051665 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 04 01:24:14.073629 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:14.052654 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": context deadline exceeded" Nov 04 01:24:14.153071 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:14.153028 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:24:14.224592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:14.216643 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:14.235604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:14.235222 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:14.407412 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:14.407319 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:14.477618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:14.428630 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:24:14.477618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:14.429041 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:24:14.537590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:14.512654 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:24:14.537590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:14.516193 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:14.571620 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:14.544653 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:14.571620 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:14.544733 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" podUID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:14.710821 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:14.709669 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:14.750829 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:14.750776 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:14.802918 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:14.802838 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 04 01:24:15.043590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:15.042661 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:15.070327 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:15.070283 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 04 01:24:15.153758 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:15.153703 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:15.217131 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:15.217077 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:15.414769 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:15.414672 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 04 01:24:15.430066 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:15.430024 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:15.431300 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:15.431270 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:15.517592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:15.514676 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:15.710056 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:15.709968 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:15.731187 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:15.731144 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:15.766674 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:15.765662 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:15.805702 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:15.804697 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.047320 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:16.047281 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.072292 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:16.072243 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.158689 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:16.158645 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.220726 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:16.220687 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.421480 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:16.420934 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.432475 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:16.431653 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:24:16.432475 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:16.432121 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:24:16.453280 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:16.453255 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:24:16.540336 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:16.540297 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.721520 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:16.721406 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.766387 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:16.766340 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.868614 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:16.858674 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 04 01:24:16.921215 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:16.921185 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:24:17.044751 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:17.044714 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:24:17.066585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:17.056651 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:24:17.066585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:17.057427 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:24:17.066585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:17.057480 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:24:17.074671 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:17.074634 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 04 01:24:17.118223 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:17.118112 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:24:17.122594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:17.120258 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:24:17.141030 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:17.135542 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:24:17.193679 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:17.193647 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:24:17.249347 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:17.249320 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:24:17.250602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:17.250554 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:24:18.313478 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:18.313445 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:18.442688 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:18.442645 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11017-7c57c6" event={"ID":"8819a827-3fc1-44d0-93db-03287c4ee9ee","Type":"ContainerStarted","Data":"b9b44d8872a777344accba54cd4ce936d92c727d2a9d8011fe3172e5c9cd3ccb"} Nov 04 01:24:18.463434 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:18.463380 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11017-7c57c6" podStartSLOduration=2.173837811 podStartE2EDuration="14.463367601s" totalImagesPullingTime="12.28952979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:18.462722589 +0000 UTC m=+1540.577832701" watchObservedRunningTime="2026-11-04 01:24:18.463367601 +0000 UTC m=+1540.578477719" Nov 04 01:24:19.537821 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:19.537789 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:19.574768 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:19.574732 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11017-7c57c6"] Nov 04 01:24:19.592292 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:19.592261 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11017-7c57c6"] Nov 04 01:24:19.767823 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:19.767791 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11017-f475f9"] Nov 04 01:24:19.768451 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:19.768431 4852 cpu_manager.go:414] "containerMap: removing container" podUID="8819a827-3fc1-44d0-93db-03287c4ee9ee" containerName="test-curl-11017-7c57c6" Nov 04 01:24:19.768586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:19.768452 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8819a827-3fc1-44d0-93db-03287c4ee9ee" containerName="test-curl-11017-7c57c6" Nov 04 01:24:19.768651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:19.768615 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="8819a827-3fc1-44d0-93db-03287c4ee9ee" containerName="test-curl-11017-7c57c6" Nov 04 01:24:19.772119 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:19.772099 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11017-f475f9" Nov 04 01:24:19.783040 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:19.783021 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11017-f475f9"] Nov 04 01:24:19.898195 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:19.898114 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhh7s\" (UniqueName: \"kubernetes.io/projected/a9b796aa-2403-4442-9c98-25e1b1d6cc7d-kube-api-access-jhh7s\") pod \"test-curl-11017-f475f9\" (UID: \"a9b796aa-2403-4442-9c98-25e1b1d6cc7d\") " pod="opendatahub/test-curl-11017-f475f9" Nov 04 01:24:19.999057 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:19.999030 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhh7s\" (UniqueName: \"kubernetes.io/projected/a9b796aa-2403-4442-9c98-25e1b1d6cc7d-kube-api-access-jhh7s\") pod \"test-curl-11017-f475f9\" (UID: \"a9b796aa-2403-4442-9c98-25e1b1d6cc7d\") " pod="opendatahub/test-curl-11017-f475f9" Nov 04 01:24:20.015167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:20.015133 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhh7s\" (UniqueName: \"kubernetes.io/projected/a9b796aa-2403-4442-9c98-25e1b1d6cc7d-kube-api-access-jhh7s\") pod \"test-curl-11017-f475f9\" (UID: \"a9b796aa-2403-4442-9c98-25e1b1d6cc7d\") " pod="opendatahub/test-curl-11017-f475f9" Nov 04 01:24:20.085176 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:20.085151 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11017-f475f9" Nov 04 01:24:20.388658 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:20.388630 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11017-f475f9"] Nov 04 01:24:20.389844 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:24:20.389821 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9b796aa_2403_4442_9c98_25e1b1d6cc7d.slice/crio-4794d12855de69b25e09165b718d736c1f78de5585e17bf3bffaf823e0773606 WatchSource:0}: Error finding container 4794d12855de69b25e09165b718d736c1f78de5585e17bf3bffaf823e0773606: Status 404 returned error can't find the container with id 4794d12855de69b25e09165b718d736c1f78de5585e17bf3bffaf823e0773606 Nov 04 01:24:20.455651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:20.455501 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11017-f475f9" event={"ID":"a9b796aa-2403-4442-9c98-25e1b1d6cc7d","Type":"ContainerStarted","Data":"4794d12855de69b25e09165b718d736c1f78de5585e17bf3bffaf823e0773606"} Nov 04 01:24:20.455731 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:20.455704 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11017-7c57c6" podUID="8819a827-3fc1-44d0-93db-03287c4ee9ee" containerName="test-curl-11017-7c57c6" containerID="cri-o://b9b44d8872a777344accba54cd4ce936d92c727d2a9d8011fe3172e5c9cd3ccb" gracePeriod=2 Nov 04 01:24:21.080767 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:21.080738 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:21.473756 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:21.473668 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11017-f475f9" event={"ID":"a9b796aa-2403-4442-9c98-25e1b1d6cc7d","Type":"ContainerStarted","Data":"2b9a2f31269cc4d0079ccd4aa148995df07ded2353610c78275db5bb68b0ea18"} Nov 04 01:24:21.899772 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:21.899730 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:22.549337 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:22.549278 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11017-f475f9" podStartSLOduration=3.54926447 podStartE2EDuration="3.54926447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:21.503499149 +0000 UTC m=+1543.618609268" watchObservedRunningTime="2026-11-04 01:24:22.54926447 +0000 UTC m=+1544.664374588" Nov 04 01:24:22.549791 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:22.549622 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11017-f475f9"] Nov 04 01:24:22.568272 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:22.568238 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11017-f475f9"] Nov 04 01:24:22.654814 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:22.654783 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11017-7c57c6" Nov 04 01:24:22.740092 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:22.740012 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8819a827-3fc1-44d0-93db-03287c4ee9ee-kube-api-access-r7rt4\" (UniqueName: \"kubernetes.io/projected/8819a827-3fc1-44d0-93db-03287c4ee9ee-kube-api-access-r7rt4\") pod \"8819a827-3fc1-44d0-93db-03287c4ee9ee\" (UID: \"8819a827-3fc1-44d0-93db-03287c4ee9ee\") " Nov 04 01:24:22.742209 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:22.742177 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8819a827-3fc1-44d0-93db-03287c4ee9ee-kube-api-access-r7rt4" pod "8819a827-3fc1-44d0-93db-03287c4ee9ee" (UID: "8819a827-3fc1-44d0-93db-03287c4ee9ee"). InnerVolumeSpecName "kube-api-access-r7rt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:22.841773 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:22.841742 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7rt4\" (UniqueName: \"kubernetes.io/projected/8819a827-3fc1-44d0-93db-03287c4ee9ee-kube-api-access-r7rt4\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:23.282598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:23.282243 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:23.486423 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:23.486387 4852 generic.go:356] "Generic (PLEG): container finished" podID="8819a827-3fc1-44d0-93db-03287c4ee9ee" containerID="b9b44d8872a777344accba54cd4ce936d92c727d2a9d8011fe3172e5c9cd3ccb" exitCode=137 Nov 04 01:24:23.486423 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:23.486425 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11017-7c57c6" Nov 04 01:24:23.486695 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:23.486487 4852 scope.go:122] "RemoveContainer" containerID="b9b44d8872a777344accba54cd4ce936d92c727d2a9d8011fe3172e5c9cd3ccb" Nov 04 01:24:23.486761 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:23.486714 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11017-f475f9" podUID="a9b796aa-2403-4442-9c98-25e1b1d6cc7d" containerName="test-curl-11017-f475f9" containerID="cri-o://2b9a2f31269cc4d0079ccd4aa148995df07ded2353610c78275db5bb68b0ea18" gracePeriod=2 Nov 04 01:24:23.491031 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:24:23.490997 4852 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11017-f475f9\" is forbidden: User \"system:node:ip-10-0-140-241.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-241.us-east-2.compute.internal' and this object" podUID="a9b796aa-2403-4442-9c98-25e1b1d6cc7d" pod="opendatahub/test-curl-11017-f475f9" Nov 04 01:24:23.496743 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:23.496728 4852 scope.go:122] "RemoveContainer" containerID="b9b44d8872a777344accba54cd4ce936d92c727d2a9d8011fe3172e5c9cd3ccb" Nov 04 01:24:23.497009 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:24:23.496992 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9b44d8872a777344accba54cd4ce936d92c727d2a9d8011fe3172e5c9cd3ccb\": container with ID starting with b9b44d8872a777344accba54cd4ce936d92c727d2a9d8011fe3172e5c9cd3ccb not found: ID does not exist" containerID="b9b44d8872a777344accba54cd4ce936d92c727d2a9d8011fe3172e5c9cd3ccb" Nov 04 01:24:23.497056 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:23.497018 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b44d8872a777344accba54cd4ce936d92c727d2a9d8011fe3172e5c9cd3ccb"} err="failed to get container status \"b9b44d8872a777344accba54cd4ce936d92c727d2a9d8011fe3172e5c9cd3ccb\": rpc error: code = NotFound desc = could not find container \"b9b44d8872a777344accba54cd4ce936d92c727d2a9d8011fe3172e5c9cd3ccb\": container with ID starting with b9b44d8872a777344accba54cd4ce936d92c727d2a9d8011fe3172e5c9cd3ccb not found: ID does not exist" Nov 04 01:24:23.499857 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:24:23.499835 4852 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11017-7c57c6\" is forbidden: User \"system:node:ip-10-0-140-241.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-241.us-east-2.compute.internal' and this object" podUID="8819a827-3fc1-44d0-93db-03287c4ee9ee" pod="opendatahub/test-curl-11017-7c57c6" Nov 04 01:24:23.502425 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:24:23.502404 4852 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11017-f475f9\" is forbidden: User \"system:node:ip-10-0-140-241.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-241.us-east-2.compute.internal' and this object" podUID="a9b796aa-2403-4442-9c98-25e1b1d6cc7d" pod="opendatahub/test-curl-11017-f475f9" Nov 04 01:24:23.581609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:23.581444 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:23.823521 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:23.823476 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:23.994365 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:23.994285 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8819a827-3fc1-44d0-93db-03287c4ee9ee" path="/var/lib/kubelet/pods/8819a827-3fc1-44d0-93db-03287c4ee9ee/volumes" Nov 04 01:24:24.214345 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:24.214274 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:24.605995 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:24.605966 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:25.409616 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:25.404113 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:25.687201 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:25.687171 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11017-f475f9" Nov 04 01:24:25.793000 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:25.792977 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b796aa-2403-4442-9c98-25e1b1d6cc7d-kube-api-access-jhh7s\" (UniqueName: \"kubernetes.io/projected/a9b796aa-2403-4442-9c98-25e1b1d6cc7d-kube-api-access-jhh7s\") pod \"a9b796aa-2403-4442-9c98-25e1b1d6cc7d\" (UID: \"a9b796aa-2403-4442-9c98-25e1b1d6cc7d\") " Nov 04 01:24:25.795210 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:25.795184 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b796aa-2403-4442-9c98-25e1b1d6cc7d-kube-api-access-jhh7s" pod "a9b796aa-2403-4442-9c98-25e1b1d6cc7d" (UID: "a9b796aa-2403-4442-9c98-25e1b1d6cc7d"). InnerVolumeSpecName "kube-api-access-jhh7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:25.895022 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:25.894995 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhh7s\" (UniqueName: \"kubernetes.io/projected/a9b796aa-2403-4442-9c98-25e1b1d6cc7d-kube-api-access-jhh7s\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:25.995130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:25.995068 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b796aa-2403-4442-9c98-25e1b1d6cc7d" path="/var/lib/kubelet/pods/a9b796aa-2403-4442-9c98-25e1b1d6cc7d/volumes" Nov 04 01:24:26.305935 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:26.305905 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:26.546352 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:26.546315 4852 generic.go:356] "Generic (PLEG): container finished" podID="a9b796aa-2403-4442-9c98-25e1b1d6cc7d" containerID="2b9a2f31269cc4d0079ccd4aa148995df07ded2353610c78275db5bb68b0ea18" exitCode=137 Nov 04 01:24:26.546531 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:26.546414 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11017-f475f9" Nov 04 01:24:26.546531 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:26.546465 4852 scope.go:122] "RemoveContainer" containerID="2b9a2f31269cc4d0079ccd4aa148995df07ded2353610c78275db5bb68b0ea18" Nov 04 01:24:26.555166 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:26.555146 4852 scope.go:122] "RemoveContainer" containerID="2b9a2f31269cc4d0079ccd4aa148995df07ded2353610c78275db5bb68b0ea18" Nov 04 01:24:26.555420 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:24:26.555399 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b9a2f31269cc4d0079ccd4aa148995df07ded2353610c78275db5bb68b0ea18\": container with ID starting with 2b9a2f31269cc4d0079ccd4aa148995df07ded2353610c78275db5bb68b0ea18 not found: ID does not exist" containerID="2b9a2f31269cc4d0079ccd4aa148995df07ded2353610c78275db5bb68b0ea18" Nov 04 01:24:26.555492 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:26.555431 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b9a2f31269cc4d0079ccd4aa148995df07ded2353610c78275db5bb68b0ea18"} err="failed to get container status \"2b9a2f31269cc4d0079ccd4aa148995df07ded2353610c78275db5bb68b0ea18\": rpc error: code = NotFound desc = could not find container \"2b9a2f31269cc4d0079ccd4aa148995df07ded2353610c78275db5bb68b0ea18\": container with ID starting with 2b9a2f31269cc4d0079ccd4aa148995df07ded2353610c78275db5bb68b0ea18 not found: ID does not exist" Nov 04 01:24:27.425223 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:27.425192 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:28.205587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:28.202356 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:28.998150 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:28.998113 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:29.294509 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.294480 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:29.589319 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.589235 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:29.634864 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.634835 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt"] Nov 04 01:24:29.635424 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.635410 4852 cpu_manager.go:414] "containerMap: removing container" podUID="a9b796aa-2403-4442-9c98-25e1b1d6cc7d" containerName="test-curl-11017-f475f9" Nov 04 01:24:29.635483 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.635426 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b796aa-2403-4442-9c98-25e1b1d6cc7d" containerName="test-curl-11017-f475f9" Nov 04 01:24:29.635543 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.635534 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b796aa-2403-4442-9c98-25e1b1d6cc7d" containerName="test-curl-11017-f475f9" Nov 04 01:24:29.640346 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.640318 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.662027 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.662004 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d92c65cb-openshift-default-dockercfg-c5sbf\"" Nov 04 01:24:29.676062 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.676023 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt"] Nov 04 01:24:29.762809 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.762780 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-credential-socket\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.762990 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.762865 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84493fd-2664-4f56-8542-84072c8850e2-istio-token\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.762990 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.762897 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84493fd-2664-4f56-8542-84072c8850e2-istiod-ca-cert\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.762990 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.762931 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-istio-data\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.763157 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.763044 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84493fd-2664-4f56-8542-84072c8850e2-istio-podinfo\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.763157 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.763093 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-workload-certs\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.763157 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.763130 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-workload-socket\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.763310 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.763201 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-istio-envoy\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.763310 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.763246 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpp5p\" (UniqueName: \"kubernetes.io/projected/d84493fd-2664-4f56-8542-84072c8850e2-kube-api-access-dpp5p\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.868077 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.867986 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84493fd-2664-4f56-8542-84072c8850e2-istio-podinfo\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.868614 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.868549 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-workload-certs\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.868614 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.868616 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-workload-socket\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.868816 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.868693 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-istio-envoy\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.868816 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.868741 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpp5p\" (UniqueName: \"kubernetes.io/projected/d84493fd-2664-4f56-8542-84072c8850e2-kube-api-access-dpp5p\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.868927 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.868856 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-credential-socket\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.868927 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.868873 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-workload-certs\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.869147 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.868945 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84493fd-2664-4f56-8542-84072c8850e2-istio-token\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.869147 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.868950 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-workload-socket\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.869147 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.868969 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84493fd-2664-4f56-8542-84072c8850e2-istiod-ca-cert\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.869147 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.869018 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-istio-data\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.869357 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.869158 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-credential-socket\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.869357 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.869239 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-istio-data\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.870126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.870098 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84493fd-2664-4f56-8542-84072c8850e2-istiod-ca-cert\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.874315 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.874286 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84493fd-2664-4f56-8542-84072c8850e2-istio-podinfo\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.877182 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.877162 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-istio-envoy\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.884049 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:24:29.884001 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:24:29.884998 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.884305 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84493fd-2664-4f56-8542-84072c8850e2-istio-token\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.884998 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.884758 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpp5p\" (UniqueName: \"kubernetes.io/projected/d84493fd-2664-4f56-8542-84072c8850e2-kube-api-access-dpp5p\") pod \"e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:29.887689 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.887670 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:29.953986 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:29.953957 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:30.148710 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:30.137304 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:30.491450 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:30.491420 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt"] Nov 04 01:24:30.491599 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:24:30.491465 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd84493fd_2664_4f56_8542_84072c8850e2.slice/crio-5559104f022dd164974d3374e536540d95c13e56914554ba6c16e6132c12dd47 WatchSource:0}: Error finding container 5559104f022dd164974d3374e536540d95c13e56914554ba6c16e6132c12dd47: Status 404 returned error can't find the container with id 5559104f022dd164974d3374e536540d95c13e56914554ba6c16e6132c12dd47 Nov 04 01:24:30.494768 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:30.494733 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:24:30.494875 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:30.494816 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:24:30.494875 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:30.494860 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:24:30.575302 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:30.575271 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" event={"ID":"d84493fd-2664-4f56-8542-84072c8850e2","Type":"ContainerStarted","Data":"8887f68abf5e204265e5753c043f656e3dea5adf66b1c098186549a8449239d8"} Nov 04 01:24:30.575411 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:30.575336 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" event={"ID":"d84493fd-2664-4f56-8542-84072c8850e2","Type":"ContainerStarted","Data":"5559104f022dd164974d3374e536540d95c13e56914554ba6c16e6132c12dd47"} Nov 04 01:24:30.614581 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:30.613070 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" podStartSLOduration=1.613052733 podStartE2EDuration="1.613052733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:30.612070223 +0000 UTC m=+1552.727180342" watchObservedRunningTime="2026-11-04 01:24:30.613052733 +0000 UTC m=+1552.728162852" Nov 04 01:24:30.958933 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:30.958896 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:31.010588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:31.006491 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:31.895777 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:31.895359 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:31.979589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:31.975917 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:32.207039 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:32.206951 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:32.588473 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:32.588440 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:32.589474 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:32.589452 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:24:32.665845 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:32.664918 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:33.374296 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.374262 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:33.585870 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.585841 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:33.598207 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.598179 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh"] Nov 04 01:24:33.602320 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.602304 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh" Nov 04 01:24:33.609700 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.609675 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d92c65cb\"" Nov 04 01:24:33.610121 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.610104 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d92c65cb-dockercfg-jv9b9\"" Nov 04 01:24:33.613867 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.613842 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh"] Nov 04 01:24:33.621053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.621026 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmwb5\" (UniqueName: \"kubernetes.io/projected/a84836f4-c505-422f-9adc-d96adcb96ced-kube-api-access-tmwb5\") pod \"payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh\" (UID: \"a84836f4-c505-422f-9adc-d96adcb96ced\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh" Nov 04 01:24:33.621251 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.621224 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a84836f4-c505-422f-9adc-d96adcb96ced-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh\" (UID: \"a84836f4-c505-422f-9adc-d96adcb96ced\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh" Nov 04 01:24:33.632929 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.632871 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m"] Nov 04 01:24:33.638293 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.637940 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m" Nov 04 01:24:33.639864 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.639838 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m"] Nov 04 01:24:33.729587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.728777 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmwb5\" (UniqueName: \"kubernetes.io/projected/a84836f4-c505-422f-9adc-d96adcb96ced-kube-api-access-tmwb5\") pod \"payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh\" (UID: \"a84836f4-c505-422f-9adc-d96adcb96ced\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh" Nov 04 01:24:33.729587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.728913 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a84836f4-c505-422f-9adc-d96adcb96ced-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh\" (UID: \"a84836f4-c505-422f-9adc-d96adcb96ced\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh" Nov 04 01:24:33.740597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.738400 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a84836f4-c505-422f-9adc-d96adcb96ced-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh\" (UID: \"a84836f4-c505-422f-9adc-d96adcb96ced\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh" Nov 04 01:24:33.773158 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.773126 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:33.777588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.774216 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmwb5\" (UniqueName: \"kubernetes.io/projected/a84836f4-c505-422f-9adc-d96adcb96ced-kube-api-access-tmwb5\") pod \"payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh\" (UID: \"a84836f4-c505-422f-9adc-d96adcb96ced\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh" Nov 04 01:24:33.831674 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.831629 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594f2fcb-8a9c-446e-bda1-7d320b1a7e6f-plugins-config-volume\") pod \"payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m\" (UID: \"594f2fcb-8a9c-446e-bda1-7d320b1a7e6f\") " pod="openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m" Nov 04 01:24:33.832072 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.832052 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5glr2\" (UniqueName: \"kubernetes.io/projected/594f2fcb-8a9c-446e-bda1-7d320b1a7e6f-kube-api-access-5glr2\") pod \"payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m\" (UID: \"594f2fcb-8a9c-446e-bda1-7d320b1a7e6f\") " pod="openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m" Nov 04 01:24:33.927950 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.927868 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh" Nov 04 01:24:33.968248 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.957794 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594f2fcb-8a9c-446e-bda1-7d320b1a7e6f-plugins-config-volume\") pod \"payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m\" (UID: \"594f2fcb-8a9c-446e-bda1-7d320b1a7e6f\") " pod="openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m" Nov 04 01:24:33.968248 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.957899 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5glr2\" (UniqueName: \"kubernetes.io/projected/594f2fcb-8a9c-446e-bda1-7d320b1a7e6f-kube-api-access-5glr2\") pod \"payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m\" (UID: \"594f2fcb-8a9c-446e-bda1-7d320b1a7e6f\") " pod="openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m" Nov 04 01:24:33.968248 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:33.959515 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594f2fcb-8a9c-446e-bda1-7d320b1a7e6f-plugins-config-volume\") pod \"payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m\" (UID: \"594f2fcb-8a9c-446e-bda1-7d320b1a7e6f\") " pod="openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m" Nov 04 01:24:34.008940 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.000020 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:34.048044 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.034854 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5glr2\" (UniqueName: \"kubernetes.io/projected/594f2fcb-8a9c-446e-bda1-7d320b1a7e6f-kube-api-access-5glr2\") pod \"payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m\" (UID: \"594f2fcb-8a9c-446e-bda1-7d320b1a7e6f\") " pod="openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m" Nov 04 01:24:34.177621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.177517 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:34.255254 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.255219 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m" Nov 04 01:24:34.363685 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.363622 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:34.376616 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.376313 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh"] Nov 04 01:24:34.551356 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.551310 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:34.553592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.553552 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m"] Nov 04 01:24:34.554323 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:24:34.554303 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594f2fcb_8a9c_446e_bda1_7d320b1a7e6f.slice/crio-229cbc375435ef81133dbfc2b2caeb389b6a1bee1bf1bdc32c7d5781fbc28929 WatchSource:0}: Error finding container 229cbc375435ef81133dbfc2b2caeb389b6a1bee1bf1bdc32c7d5781fbc28929: Status 404 returned error can't find the container with id 229cbc375435ef81133dbfc2b2caeb389b6a1bee1bf1bdc32c7d5781fbc28929 Nov 04 01:24:34.604058 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.604029 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m" event={"ID":"594f2fcb-8a9c-446e-bda1-7d320b1a7e6f","Type":"ContainerStarted","Data":"229cbc375435ef81133dbfc2b2caeb389b6a1bee1bf1bdc32c7d5781fbc28929"} Nov 04 01:24:34.605771 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.605734 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh" event={"ID":"a84836f4-c505-422f-9adc-d96adcb96ced","Type":"ContainerStarted","Data":"5451bf1336b132e54e8d2b5a1e2d822c6f85bad1d952f4a106cb3a6439a23437"} Nov 04 01:24:34.605924 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.605790 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh" event={"ID":"a84836f4-c505-422f-9adc-d96adcb96ced","Type":"ContainerStarted","Data":"25657dc0c7890d7a96deaf7c956badb016089323eb5b07f639d9f92bca795f7e"} Nov 04 01:24:34.606200 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.606158 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh" Nov 04 01:24:34.684315 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.683829 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh" podStartSLOduration=1.683809191 podStartE2EDuration="1.683809191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:34.660026043 +0000 UTC m=+1556.775136164" watchObservedRunningTime="2026-11-04 01:24:34.683809191 +0000 UTC m=+1556.798919310" Nov 04 01:24:34.778690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.778659 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6"] Nov 04 01:24:34.786882 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.786857 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6" Nov 04 01:24:34.801473 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.799803 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6"] Nov 04 01:24:34.803656 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.803626 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-7b10de73\"" Nov 04 01:24:34.854829 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.854794 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fchvv\" (UniqueName: \"kubernetes.io/projected/3c9e69da-ee72-4330-b006-baa850a853d9-kube-api-access-fchvv\") pod \"payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6\" (UID: \"3c9e69da-ee72-4330-b006-baa850a853d9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6" Nov 04 01:24:34.855120 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.855100 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9e69da-ee72-4330-b006-baa850a853d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6\" (UID: \"3c9e69da-ee72-4330-b006-baa850a853d9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6" Nov 04 01:24:34.956273 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.956222 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9e69da-ee72-4330-b006-baa850a853d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6\" (UID: \"3c9e69da-ee72-4330-b006-baa850a853d9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6" Nov 04 01:24:34.956457 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.956408 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fchvv\" (UniqueName: \"kubernetes.io/projected/3c9e69da-ee72-4330-b006-baa850a853d9-kube-api-access-fchvv\") pod \"payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6\" (UID: \"3c9e69da-ee72-4330-b006-baa850a853d9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6" Nov 04 01:24:34.957476 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.957446 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9e69da-ee72-4330-b006-baa850a853d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6\" (UID: \"3c9e69da-ee72-4330-b006-baa850a853d9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6" Nov 04 01:24:34.980732 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:34.980704 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fchvv\" (UniqueName: \"kubernetes.io/projected/3c9e69da-ee72-4330-b006-baa850a853d9-kube-api-access-fchvv\") pod \"payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6\" (UID: \"3c9e69da-ee72-4330-b006-baa850a853d9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6" Nov 04 01:24:35.140643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:35.133870 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6" Nov 04 01:24:35.614219 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:35.614175 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m" event={"ID":"594f2fcb-8a9c-446e-bda1-7d320b1a7e6f","Type":"ContainerStarted","Data":"3e36731026f255d034b4814c3ae28b3dc6d6971a2e7d99dd6350fa3bc8487162"} Nov 04 01:24:35.614649 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:35.614288 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m" Nov 04 01:24:35.635540 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:35.635483 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m" podStartSLOduration=2.635465193 podStartE2EDuration="2.635465193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:35.633986399 +0000 UTC m=+1557.749096517" watchObservedRunningTime="2026-11-04 01:24:35.635465193 +0000 UTC m=+1557.750575312" Nov 04 01:24:35.702896 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:35.702868 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6"] Nov 04 01:24:35.703599 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:24:35.703539 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9e69da_ee72_4330_b006_baa850a853d9.slice/crio-a3f202fa1ba6802bcff42c559594737dbea02aa22d7c69e1d365066958c11e72 WatchSource:0}: Error finding container a3f202fa1ba6802bcff42c559594737dbea02aa22d7c69e1d365066958c11e72: Status 404 returned error can't find the container with id a3f202fa1ba6802bcff42c559594737dbea02aa22d7c69e1d365066958c11e72 Nov 04 01:24:35.958590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:35.958280 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:36.117394 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:36.117360 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:36.623630 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:36.623553 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6" event={"ID":"3c9e69da-ee72-4330-b006-baa850a853d9","Type":"ContainerStarted","Data":"40ca46b848f87ec684fa02149ee037adbafb1ba7da0c96b45ace7887f571ccc4"} Nov 04 01:24:36.624067 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:36.623636 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6" event={"ID":"3c9e69da-ee72-4330-b006-baa850a853d9","Type":"ContainerStarted","Data":"a3f202fa1ba6802bcff42c559594737dbea02aa22d7c69e1d365066958c11e72"} Nov 04 01:24:36.624067 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:36.623827 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6" Nov 04 01:24:36.645050 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:36.645004 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6" podStartSLOduration=2.644991673 podStartE2EDuration="2.644991673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:36.642472423 +0000 UTC m=+1558.757582543" watchObservedRunningTime="2026-11-04 01:24:36.644991673 +0000 UTC m=+1558.760101790" Nov 04 01:24:36.740621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:36.740581 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:37.735208 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:37.735132 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:38.201116 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:38.201086 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:38.342211 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:38.342186 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:38.626592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:38.626010 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:39.110410 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.110381 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:39.271726 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.271698 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:39.445640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.445522 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:39.480428 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.480404 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5"] Nov 04 01:24:39.487632 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.487609 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5" Nov 04 01:24:39.497271 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.497250 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5"] Nov 04 01:24:39.498879 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.498858 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d7024f0c\"" Nov 04 01:24:39.499075 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.499059 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d7024f0c-dockercfg-m8f46\"" Nov 04 01:24:39.501936 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.501910 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd"] Nov 04 01:24:39.507610 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.507593 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd" Nov 04 01:24:39.524744 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.524059 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd"] Nov 04 01:24:39.818590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.809267 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:39.864142 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.864045 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc908b7-78ae-44d2-a916-598712dc3a72-plugins-config-volume\") pod \"payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd\" (UID: \"3cc908b7-78ae-44d2-a916-598712dc3a72\") " pod="openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd" Nov 04 01:24:39.864867 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.864839 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks29l\" (UniqueName: \"kubernetes.io/projected/bfb9e9da-63a3-4c80-aca8-7600e83ee4dd-kube-api-access-ks29l\") pod \"payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5\" (UID: \"bfb9e9da-63a3-4c80-aca8-7600e83ee4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5" Nov 04 01:24:39.865867 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.865174 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-768w6\" (UniqueName: \"kubernetes.io/projected/3cc908b7-78ae-44d2-a916-598712dc3a72-kube-api-access-768w6\") pod \"payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd\" (UID: \"3cc908b7-78ae-44d2-a916-598712dc3a72\") " pod="openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd" Nov 04 01:24:39.866192 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.866169 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb9e9da-63a3-4c80-aca8-7600e83ee4dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5\" (UID: \"bfb9e9da-63a3-4c80-aca8-7600e83ee4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5" Nov 04 01:24:39.967958 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.967925 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc908b7-78ae-44d2-a916-598712dc3a72-plugins-config-volume\") pod \"payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd\" (UID: \"3cc908b7-78ae-44d2-a916-598712dc3a72\") " pod="openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd" Nov 04 01:24:39.968155 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.967994 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks29l\" (UniqueName: \"kubernetes.io/projected/bfb9e9da-63a3-4c80-aca8-7600e83ee4dd-kube-api-access-ks29l\") pod \"payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5\" (UID: \"bfb9e9da-63a3-4c80-aca8-7600e83ee4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5" Nov 04 01:24:39.968155 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.968093 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-768w6\" (UniqueName: \"kubernetes.io/projected/3cc908b7-78ae-44d2-a916-598712dc3a72-kube-api-access-768w6\") pod \"payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd\" (UID: \"3cc908b7-78ae-44d2-a916-598712dc3a72\") " pod="openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd" Nov 04 01:24:39.968155 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.968118 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb9e9da-63a3-4c80-aca8-7600e83ee4dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5\" (UID: \"bfb9e9da-63a3-4c80-aca8-7600e83ee4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5" Nov 04 01:24:39.968750 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.968724 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc908b7-78ae-44d2-a916-598712dc3a72-plugins-config-volume\") pod \"payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd\" (UID: \"3cc908b7-78ae-44d2-a916-598712dc3a72\") " pod="openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd" Nov 04 01:24:39.968919 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.968897 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb9e9da-63a3-4c80-aca8-7600e83ee4dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5\" (UID: \"bfb9e9da-63a3-4c80-aca8-7600e83ee4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5" Nov 04 01:24:39.971101 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.971080 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:39.981715 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.981692 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-768w6\" (UniqueName: \"kubernetes.io/projected/3cc908b7-78ae-44d2-a916-598712dc3a72-kube-api-access-768w6\") pod \"payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd\" (UID: \"3cc908b7-78ae-44d2-a916-598712dc3a72\") " pod="openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd" Nov 04 01:24:39.982251 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:39.982228 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks29l\" (UniqueName: \"kubernetes.io/projected/bfb9e9da-63a3-4c80-aca8-7600e83ee4dd-kube-api-access-ks29l\") pod \"payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5\" (UID: \"bfb9e9da-63a3-4c80-aca8-7600e83ee4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5" Nov 04 01:24:40.105392 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.105311 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5" Nov 04 01:24:40.128506 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.127243 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd" Nov 04 01:24:40.394103 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.394065 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5"] Nov 04 01:24:40.394237 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:24:40.394202 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb9e9da_63a3_4c80_aca8_7600e83ee4dd.slice/crio-28d3c52adabfbf875f9980c9030ef1785f8523ebeb33a4c4bf22cea047202a5c WatchSource:0}: Error finding container 28d3c52adabfbf875f9980c9030ef1785f8523ebeb33a4c4bf22cea047202a5c: Status 404 returned error can't find the container with id 28d3c52adabfbf875f9980c9030ef1785f8523ebeb33a4c4bf22cea047202a5c Nov 04 01:24:40.410792 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.410760 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk"] Nov 04 01:24:40.444941 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.444913 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk"] Nov 04 01:24:40.445076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.444952 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd"] Nov 04 01:24:40.445076 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:24:40.444972 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc908b7_78ae_44d2_a916_598712dc3a72.slice/crio-b510f6e01f69995a4fe0d79b47b78667b88f01315d1456a9b35524915dca1176 WatchSource:0}: Error finding container b510f6e01f69995a4fe0d79b47b78667b88f01315d1456a9b35524915dca1176: Status 404 returned error can't find the container with id b510f6e01f69995a4fe0d79b47b78667b88f01315d1456a9b35524915dca1176 Nov 04 01:24:40.445076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.445028 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" Nov 04 01:24:40.456685 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.456664 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d92c65cb-serving-cert\"" Nov 04 01:24:40.474940 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.474908 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twfnh\" (UniqueName: \"kubernetes.io/projected/78a2fbe6-6fca-428a-9bf6-43bb549282ff-kube-api-access-twfnh\") pod \"maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk\" (UID: \"78a2fbe6-6fca-428a-9bf6-43bb549282ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" Nov 04 01:24:40.475074 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.474988 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a2fbe6-6fca-428a-9bf6-43bb549282ff-openshift-service-ca\") pod \"maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk\" (UID: \"78a2fbe6-6fca-428a-9bf6-43bb549282ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" Nov 04 01:24:40.475074 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.475031 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78a2fbe6-6fca-428a-9bf6-43bb549282ff-maas-api-tls\") pod \"maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk\" (UID: \"78a2fbe6-6fca-428a-9bf6-43bb549282ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" Nov 04 01:24:40.475074 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.475069 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78a2fbe6-6fca-428a-9bf6-43bb549282ff-metrics-certs\") pod \"maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk\" (UID: \"78a2fbe6-6fca-428a-9bf6-43bb549282ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" Nov 04 01:24:40.542525 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.542498 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:40.598603 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.587591 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twfnh\" (UniqueName: \"kubernetes.io/projected/78a2fbe6-6fca-428a-9bf6-43bb549282ff-kube-api-access-twfnh\") pod \"maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk\" (UID: \"78a2fbe6-6fca-428a-9bf6-43bb549282ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" Nov 04 01:24:40.598603 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.587668 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a2fbe6-6fca-428a-9bf6-43bb549282ff-openshift-service-ca\") pod \"maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk\" (UID: \"78a2fbe6-6fca-428a-9bf6-43bb549282ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" Nov 04 01:24:40.598603 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.587716 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78a2fbe6-6fca-428a-9bf6-43bb549282ff-maas-api-tls\") pod \"maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk\" (UID: \"78a2fbe6-6fca-428a-9bf6-43bb549282ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" Nov 04 01:24:40.598603 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.587755 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78a2fbe6-6fca-428a-9bf6-43bb549282ff-metrics-certs\") pod \"maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk\" (UID: \"78a2fbe6-6fca-428a-9bf6-43bb549282ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" Nov 04 01:24:40.604621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.600818 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a2fbe6-6fca-428a-9bf6-43bb549282ff-openshift-service-ca\") pod \"maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk\" (UID: \"78a2fbe6-6fca-428a-9bf6-43bb549282ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" Nov 04 01:24:40.604621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.603581 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78a2fbe6-6fca-428a-9bf6-43bb549282ff-metrics-certs\") pod \"maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk\" (UID: \"78a2fbe6-6fca-428a-9bf6-43bb549282ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" Nov 04 01:24:40.625591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.623258 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78a2fbe6-6fca-428a-9bf6-43bb549282ff-maas-api-tls\") pod \"maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk\" (UID: \"78a2fbe6-6fca-428a-9bf6-43bb549282ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" Nov 04 01:24:40.625591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.623276 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twfnh\" (UniqueName: \"kubernetes.io/projected/78a2fbe6-6fca-428a-9bf6-43bb549282ff-kube-api-access-twfnh\") pod \"maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk\" (UID: \"78a2fbe6-6fca-428a-9bf6-43bb549282ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" Nov 04 01:24:40.742891 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.742818 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:40.765469 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.765443 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" Nov 04 01:24:40.894105 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.894069 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd" event={"ID":"3cc908b7-78ae-44d2-a916-598712dc3a72","Type":"ContainerStarted","Data":"17ce10cf633b7e5688e96c68ca92e3f691a248203ab9af62c1ee907289083147"} Nov 04 01:24:40.894286 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.894134 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd" event={"ID":"3cc908b7-78ae-44d2-a916-598712dc3a72","Type":"ContainerStarted","Data":"b510f6e01f69995a4fe0d79b47b78667b88f01315d1456a9b35524915dca1176"} Nov 04 01:24:40.894345 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.894336 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd" Nov 04 01:24:40.896327 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.896299 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5" event={"ID":"bfb9e9da-63a3-4c80-aca8-7600e83ee4dd","Type":"ContainerStarted","Data":"cd5964e2f6e5564402203377277302c01ff6b5531b3ecb0da517905c3cd69772"} Nov 04 01:24:40.896454 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.896343 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5" event={"ID":"bfb9e9da-63a3-4c80-aca8-7600e83ee4dd","Type":"ContainerStarted","Data":"28d3c52adabfbf875f9980c9030ef1785f8523ebeb33a4c4bf22cea047202a5c"} Nov 04 01:24:40.896454 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.896363 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5" Nov 04 01:24:40.925392 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.925336 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd" podStartSLOduration=1.925317422 podStartE2EDuration="1.925317422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:40.917734846 +0000 UTC m=+1563.032844965" watchObservedRunningTime="2026-11-04 01:24:40.925317422 +0000 UTC m=+1563.040427540" Nov 04 01:24:40.938800 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:40.938749 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5" podStartSLOduration=1.938737063 podStartE2EDuration="1.938737063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:40.937436673 +0000 UTC m=+1563.052546788" watchObservedRunningTime="2026-11-04 01:24:40.938737063 +0000 UTC m=+1563.053847181" Nov 04 01:24:41.076939 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:41.069084 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk"] Nov 04 01:24:41.484637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:41.483707 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:41.915135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:41.915097 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" event={"ID":"78a2fbe6-6fca-428a-9bf6-43bb549282ff","Type":"ContainerStarted","Data":"e1066bb77ed1b6fb4def9d623bda2bf8eb049e70d2a70c42b05242c1ffe43adf"} Nov 04 01:24:41.915304 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:41.915173 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" event={"ID":"78a2fbe6-6fca-428a-9bf6-43bb549282ff","Type":"ContainerStarted","Data":"48e47237996a31923cf164487d2f0c9c96a2de7adfcb9bddc7e33b0f832d99b3"} Nov 04 01:24:42.396522 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:42.387168 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:42.548328 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:42.548293 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:42.922765 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:42.922734 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" Nov 04 01:24:43.288321 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:43.288291 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:44.686586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:44.685656 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:44.720591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:44.714675 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:44.857586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:44.855654 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:45.034590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:45.021116 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:45.131591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:45.130639 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:45.131591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:45.130972 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:24:45.226590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:45.219670 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:45.318648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:45.317662 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:45.362618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:45.361657 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:45.529232 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:45.528680 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:24:45.632969 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:45.631736 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh" Nov 04 01:24:45.660590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:45.656393 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" podStartSLOduration=5.030218391 podStartE2EDuration="5.656373706s" totalImagesPullingTime="626.155315ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:41.947243236 +0000 UTC m=+1564.062353353" watchObservedRunningTime="2026-11-04 01:24:45.656373706 +0000 UTC m=+1567.771483825" Nov 04 01:24:45.688439 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:45.688399 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:45.689590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:45.689543 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:24:46.335684 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:45.981965 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:24:46.335684 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:45.983442 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:24:46.335684 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:46.008922 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:24:46.335684 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:46.009141 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:24:46.335684 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:46.070831 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:46.335684 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:46.237711 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 04 01:24:46.335684 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:46.238307 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:24:46.454596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:46.442698 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:46.819860 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:46.696259 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:24:46.819860 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:46.772894 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 04 01:24:46.903591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:46.903464 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:24:46.914631 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:46.914595 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 04 01:24:46.920289 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:46.916662 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:24:46.985612 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:46.985548 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:46.986962 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:46.985943 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:47.092021 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.089484 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:47.111896 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:24:47.101688 4852 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.092s" Nov 04 01:24:47.112200 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.112182 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:24:47.112512 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.112498 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:24:47.112704 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.112680 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:24:47.112881 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.112859 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:24:47.113025 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.113011 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:24:47.113201 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.113178 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:24:47.113353 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.113340 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:24:47.113520 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.113507 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m" Nov 04 01:24:47.239678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.239636 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:47.254696 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.240311 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 04 01:24:47.534359 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.534312 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:47.659905 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.659634 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:24:47.674111 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.674081 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6" Nov 04 01:24:47.701084 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.699918 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:24:47.702608 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.701995 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:24:47.730975 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.729953 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:24:47.774117 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.773309 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 04 01:24:47.933681 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.928914 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 04 01:24:47.933681 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.929223 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": context deadline exceeded" Nov 04 01:24:47.933681 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.929545 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:47.976592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.961739 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:47.976592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:47.962116 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Nov 04 01:24:48.005735 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:48.005684 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:24:48.142640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:48.115374 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 04 01:24:48.260302 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:48.259653 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:48.260302 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:48.259995 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:24:48.516330 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:48.508354 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:48.538394 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:48.537661 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:48.606911 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:48.606661 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:48.782803 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:48.782674 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:48.930731 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:48.930684 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:48.931347 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:48.930979 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:24:48.963680 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:48.962666 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:48.999851 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:48.999768 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" Nov 04 01:24:49.006721 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:49.006689 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:49.119814 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:49.119713 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:49.283607 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:49.283538 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:24:49.290005 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:49.283655 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:49.538287 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:49.538241 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:49.661785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:49.661741 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:49.737712 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:49.737665 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:49.783715 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:49.783662 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:49.921830 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:49.921700 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:49.928355 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:49.928315 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:49.933281 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:49.932731 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 04 01:24:49.934131 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:49.934102 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:49.967703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:49.966666 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:50.020585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:50.010547 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:50.043345 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:50.041792 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:24:50.199803 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:50.199696 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:50.294955 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:50.294908 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:50.296205 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:50.296173 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:50.556594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:50.553854 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:50.573588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:50.569677 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:50.665703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:50.665656 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:24:50.666548 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:50.666524 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:24:50.992037 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:50.991923 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:50.993271 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:50.991875 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:50.994062 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:50.994040 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:51.020186 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:51.020148 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 04 01:24:51.082778 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:51.081215 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:24:51.093173 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:51.093101 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:51.117136 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:51.117081 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" podUID="a4cd0527-183f-484b-8dc5-8fc84aa45b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:51.145676 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:51.145593 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:51.216680 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:51.215664 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:51.317781 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:51.317722 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:51.415251 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:51.406682 4852 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:51.415251 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:51.406761 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" podUID="fbc69a29-d23c-4da4-a6af-be04977ee457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:51.543018 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:51.542985 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:51.558142 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:51.558097 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:51.673642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:51.672684 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:24:51.731396 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:51.731356 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:51.960023 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:51.959935 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd" Nov 04 01:24:51.994593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:51.990750 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5" Nov 04 01:24:51.995390 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:51.995348 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:51.996610 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:51.996585 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:51.997051 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:51.997027 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 04 01:24:52.025590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:52.021814 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:52.025590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:52.022189 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": context deadline exceeded" Nov 04 01:24:52.085706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:52.085650 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 04 01:24:52.168582 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:52.124630 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:24:52.169159 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:52.167658 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:52.169350 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:52.169314 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" podUID="7619e54f-752b-428f-b955-b2dec4471c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:52.217302 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:52.216333 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 04 01:24:52.281586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:52.274433 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:24:52.330023 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:52.329654 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:52.350490 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:52.350465 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:24:52.360581 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:52.352195 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:24:52.420079 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:52.420042 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:52.442635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:52.442607 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:24:52.451977 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:52.451644 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:24:52.453850 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:52.453832 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:24:52.454703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:52.454687 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:24:52.465126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:52.465111 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:24:52.467827 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:52.467735 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:52.475161 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:52.475145 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:24:52.677010 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:52.676978 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:53.600069 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:53.600031 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:54.213535 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:54.213502 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh"] Nov 04 01:24:54.218691 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:54.218670 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" Nov 04 01:24:54.232171 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:54.232130 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d7024f0c-serving-cert\"" Nov 04 01:24:54.238746 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:54.237872 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh"] Nov 04 01:24:54.355588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:54.355516 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:54.390933 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:54.390306 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s84rr\" (UniqueName: \"kubernetes.io/projected/5972b292-b0b7-45af-9674-f0c00f20c1c9-kube-api-access-s84rr\") pod \"maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh\" (UID: \"5972b292-b0b7-45af-9674-f0c00f20c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" Nov 04 01:24:54.390933 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:54.390470 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5972b292-b0b7-45af-9674-f0c00f20c1c9-openshift-service-ca\") pod \"maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh\" (UID: \"5972b292-b0b7-45af-9674-f0c00f20c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" Nov 04 01:24:54.390933 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:54.390681 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-maas-api-tls\") pod \"maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh\" (UID: \"5972b292-b0b7-45af-9674-f0c00f20c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" Nov 04 01:24:54.390933 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:54.390783 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-metrics-certs\") pod \"maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh\" (UID: \"5972b292-b0b7-45af-9674-f0c00f20c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" Nov 04 01:24:54.492389 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:54.492298 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-metrics-certs\") pod \"maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh\" (UID: \"5972b292-b0b7-45af-9674-f0c00f20c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" Nov 04 01:24:54.492389 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:54.492348 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s84rr\" (UniqueName: \"kubernetes.io/projected/5972b292-b0b7-45af-9674-f0c00f20c1c9-kube-api-access-s84rr\") pod \"maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh\" (UID: \"5972b292-b0b7-45af-9674-f0c00f20c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" Nov 04 01:24:54.492690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:54.492474 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5972b292-b0b7-45af-9674-f0c00f20c1c9-openshift-service-ca\") pod \"maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh\" (UID: \"5972b292-b0b7-45af-9674-f0c00f20c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" Nov 04 01:24:54.492690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:54.492675 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-maas-api-tls\") pod \"maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh\" (UID: \"5972b292-b0b7-45af-9674-f0c00f20c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" Nov 04 01:24:54.492883 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:24:54.492809 4852 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-serving-cert: secret "maas-api-e2e-ait-d7024f0c-serving-cert" not found Nov 04 01:24:54.492883 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:24:54.492883 4852 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-maas-api-tls podName:5972b292-b0b7-45af-9674-f0c00f20c1c9 nodeName:}" failed. No retries permitted until 2026-11-04 01:24:54.992861479 +0000 UTC m=+1577.107971578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-maas-api-tls") pod "maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" (UID: "5972b292-b0b7-45af-9674-f0c00f20c1c9") : secret "maas-api-e2e-ait-d7024f0c-serving-cert" not found Nov 04 01:24:54.493218 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:54.493197 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5972b292-b0b7-45af-9674-f0c00f20c1c9-openshift-service-ca\") pod \"maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh\" (UID: \"5972b292-b0b7-45af-9674-f0c00f20c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" Nov 04 01:24:54.495319 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:54.495300 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-metrics-certs\") pod \"maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh\" (UID: \"5972b292-b0b7-45af-9674-f0c00f20c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" Nov 04 01:24:54.504863 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:54.504838 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s84rr\" (UniqueName: \"kubernetes.io/projected/5972b292-b0b7-45af-9674-f0c00f20c1c9-kube-api-access-s84rr\") pod \"maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh\" (UID: \"5972b292-b0b7-45af-9674-f0c00f20c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" Nov 04 01:24:54.530879 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:54.530846 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:55.002731 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:55.002690 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-maas-api-tls\") pod \"maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh\" (UID: \"5972b292-b0b7-45af-9674-f0c00f20c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" Nov 04 01:24:55.003539 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:24:55.003515 4852 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-serving-cert: secret "maas-api-e2e-ait-d7024f0c-serving-cert" not found Nov 04 01:24:55.003737 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:24:55.003725 4852 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-maas-api-tls podName:5972b292-b0b7-45af-9674-f0c00f20c1c9 nodeName:}" failed. No retries permitted until 2026-11-04 01:24:56.003702844 +0000 UTC m=+1578.118812961 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-maas-api-tls") pod "maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" (UID: "5972b292-b0b7-45af-9674-f0c00f20c1c9") : secret "maas-api-e2e-ait-d7024f0c-serving-cert" not found Nov 04 01:24:55.261954 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:55.261919 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:56.032557 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:56.032510 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:56.061696 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:56.061655 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-maas-api-tls\") pod \"maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh\" (UID: \"5972b292-b0b7-45af-9674-f0c00f20c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" Nov 04 01:24:56.062280 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:24:56.062246 4852 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-serving-cert: secret "maas-api-e2e-ait-d7024f0c-serving-cert" not found Nov 04 01:24:56.062513 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:24:56.062501 4852 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-maas-api-tls podName:5972b292-b0b7-45af-9674-f0c00f20c1c9 nodeName:}" failed. No retries permitted until 2026-11-04 01:24:58.062476351 +0000 UTC m=+1580.177586467 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-maas-api-tls") pod "maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" (UID: "5972b292-b0b7-45af-9674-f0c00f20c1c9") : secret "maas-api-e2e-ait-d7024f0c-serving-cert" not found Nov 04 01:24:56.828587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:56.824954 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:57.626281 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:57.626246 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:58.091472 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:58.091439 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-maas-api-tls\") pod \"maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh\" (UID: \"5972b292-b0b7-45af-9674-f0c00f20c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" Nov 04 01:24:58.091773 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:24:58.091539 4852 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-serving-cert: secret "maas-api-e2e-ait-d7024f0c-serving-cert" not found Nov 04 01:24:58.091773 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:24:58.091642 4852 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-maas-api-tls podName:5972b292-b0b7-45af-9674-f0c00f20c1c9 nodeName:}" failed. No retries permitted until 2026-11-04 01:25:02.091625613 +0000 UTC m=+1584.206735722 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-maas-api-tls") pod "maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" (UID: "5972b292-b0b7-45af-9674-f0c00f20c1c9") : secret "maas-api-e2e-ait-d7024f0c-serving-cert" not found Nov 04 01:24:58.460759 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:58.460678 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:59.159711 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:59.159677 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:59.294145 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:24:59.294118 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:59.866513 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:24:59.866472 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:25:00.055645 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:00.055610 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:00.796269 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:00.796218 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:01.620353 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:01.613197 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:02.103729 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:02.103694 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-maas-api-tls\") pod \"maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh\" (UID: \"5972b292-b0b7-45af-9674-f0c00f20c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" Nov 04 01:25:02.106263 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:02.106234 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-maas-api-tls\") pod \"maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh\" (UID: \"5972b292-b0b7-45af-9674-f0c00f20c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" Nov 04 01:25:02.347094 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:02.347063 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" Nov 04 01:25:02.462407 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:02.462326 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:02.636124 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:25:02.636087 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5972b292_b0b7_45af_9674_f0c00f20c1c9.slice/crio-7adf7924e69b17be9f380aae0b20ef763ec9b71f7ef3ebaf3671340db231760f WatchSource:0}: Error finding container 7adf7924e69b17be9f380aae0b20ef763ec9b71f7ef3ebaf3671340db231760f: Status 404 returned error can't find the container with id 7adf7924e69b17be9f380aae0b20ef763ec9b71f7ef3ebaf3671340db231760f Nov 04 01:25:02.636468 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:02.636450 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh"] Nov 04 01:25:02.637886 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:02.637870 4852 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:25:03.469642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:03.466520 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:03.594635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:03.594600 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" event={"ID":"5972b292-b0b7-45af-9674-f0c00f20c1c9","Type":"ContainerStarted","Data":"3b28ca4b4f5964c01b126dc0576dd3e135325b25265a9f495c8875425207e790"} Nov 04 01:25:03.594817 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:03.594748 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" Nov 04 01:25:03.594817 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:03.594789 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" event={"ID":"5972b292-b0b7-45af-9674-f0c00f20c1c9","Type":"ContainerStarted","Data":"7adf7924e69b17be9f380aae0b20ef763ec9b71f7ef3ebaf3671340db231760f"} Nov 04 01:25:03.620672 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:03.620619 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" podStartSLOduration=9.151324417 podStartE2EDuration="9.620603985s" totalImagesPullingTime="469.279568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:03.618402236 +0000 UTC m=+1585.733512362" watchObservedRunningTime="2026-11-04 01:25:03.620603985 +0000 UTC m=+1585.735714102" Nov 04 01:25:04.701434 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:04.701396 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:04.895027 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:04.894997 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:05.045715 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:05.045674 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39bf412b-ph6mq"] Nov 04 01:25:05.049052 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:05.049030 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39bf412b-ph6mq"] Nov 04 01:25:05.216465 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:05.216433 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:05.673586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:05.673116 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:05.994875 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:05.994796 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:05.997412 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:05.997384 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35a44cda-acce-49f1-865e-96af8e234fd0" path="/var/lib/kubelet/pods/35a44cda-acce-49f1-865e-96af8e234fd0/volumes" Nov 04 01:25:06.448961 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:06.448927 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:07.146981 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:07.146949 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:07.880584 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:07.880532 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:08.660487 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:08.660450 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:09.581056 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:09.581020 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:09.606283 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:09.606260 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" Nov 04 01:25:10.177589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:10.173077 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:11.301273 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:11.301236 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:11.659396 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:11.659318 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:25:11.786805 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:11.786775 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:12.241829 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:12.241800 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:12.647273 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:12.647233 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:13.704591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:13.704504 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m"] Nov 04 01:25:13.829587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:13.789897 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh"] Nov 04 01:25:14.054590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:14.054429 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:14.388642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:14.387756 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:14.530184 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:14.530147 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6"] Nov 04 01:25:14.662637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:14.660662 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:14.763613 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:14.760472 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:25:14.818591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:14.814797 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 04 01:25:14.937641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:14.934073 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:14.951587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:14.948405 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:25:14.981165 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:14.948597 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 04 01:25:14.982068 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:14.982029 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:15.112717 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:15.112670 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:15.363591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:15.363167 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:15.472184 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:15.445799 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:25:15.605590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:15.582670 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:15.773941 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:15.771737 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:15.773941 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:15.772167 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:25:15.825589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:15.817705 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:15.833586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:15.830092 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:25:16.054834 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:16.054703 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:16.182817 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:16.182760 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:25:16.246599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:16.229271 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:25:16.286585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:16.230311 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:16.484648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:16.415147 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:16.484648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:16.416424 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 04 01:25:16.484648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:16.417868 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 04 01:25:16.484648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:16.433041 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:25:16.484648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:16.474739 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:25:16.666589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:16.647817 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:16.952127 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:16.948893 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:16.952127 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:16.951372 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:16.983774 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:16.972398 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:25:16.983774 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:16.972465 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:25:16.983774 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:16.972503 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:25:16.983774 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:16.972546 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:25:16.999628 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:16.984227 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:25:16.999628 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:16.984295 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:25:17.066144 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:17.066114 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:17.204226 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:17.204126 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:17.241360 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:17.211070 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:25:17.241360 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:17.204489 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:17.301261 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:17.300775 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:25:17.454893 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:17.449670 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:17.454893 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:17.449986 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 04 01:25:17.526598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:17.502181 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:25:17.586347 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:17.528659 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 04 01:25:17.587618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:17.528696 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 04 01:25:17.596959 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:17.528726 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:17.619519 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:17.619260 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": context deadline exceeded" Nov 04 01:25:17.643792 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:17.642206 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:17.651221 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:17.650716 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:17.952590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:17.951874 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:17.978589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:17.953358 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:17.978589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:17.953960 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:18.222646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:18.220375 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:25:18.289592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:18.289525 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:18.323531 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:18.323485 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:25:18.631904 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:18.631850 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:18.632377 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:18.632351 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:18.632853 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:18.632824 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:18.633243 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:18.633223 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:25:18.649591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:18.635046 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:18.907035 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:18.683682 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:18.907816 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:18.683727 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:18.908198 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:18.719656 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m" podUID="594f2fcb-8a9c-446e-bda1-7d320b1a7e6f" containerName="payload-processing" containerID="cri-o://3e36731026f255d034b4814c3ae28b3dc6d6971a2e7d99dd6350fa3bc8487162" gracePeriod=25 Nov 04 01:25:18.908454 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:18.792879 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh" podUID="a84836f4-c505-422f-9adc-d96adcb96ced" containerName="payload-pre-processing" containerID="cri-o://5451bf1336b132e54e8d2b5a1e2d822c6f85bad1d952f4a106cb3a6439a23437" gracePeriod=25 Nov 04 01:25:18.958953 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:18.957798 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:25:18.972779 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:18.972746 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 04 01:25:19.300723 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:19.272666 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:25:19.300723 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:19.299114 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:19.357725 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:19.357680 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:19.507640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:19.507610 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:25:19.609634 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:19.603641 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6" podUID="3c9e69da-ee72-4330-b006-baa850a853d9" containerName="payload-pre-processing" containerID="cri-o://40ca46b848f87ec684fa02149ee037adbafb1ba7da0c96b45ace7887f571ccc4" gracePeriod=25 Nov 04 01:25:19.642766 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:19.642580 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 04 01:25:19.645400 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:19.642962 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:19.645400 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:19.643235 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:25:19.886055 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:19.885673 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:19.942654 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:19.914019 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:19.942654 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:19.919114 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 04 01:25:19.955730 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:19.955686 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:19.961591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:19.959646 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:25:19.987236 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:19.986746 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 04 01:25:20.350612 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:20.337303 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:20.350612 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:20.337725 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:25:20.350612 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:20.338046 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": context deadline exceeded" Nov 04 01:25:20.631128 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:20.617381 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded" Nov 04 01:25:20.804609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:20.804492 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:20.857596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:20.839372 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:20.857596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:20.839741 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:20.857596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:20.839987 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:20.924785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:20.924745 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 04 01:25:20.963846 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:20.963809 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:25:20.980649 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:20.964853 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:25:21.025681 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:20.996080 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 04 01:25:21.088070 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:25:21.086415 4852 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.099s" Nov 04 01:25:21.088070 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:21.086515 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:25:21.228883 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:21.189129 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:25:21.322764 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:21.294280 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:25:21.345728 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:21.345686 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:25:21.431699 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:21.431668 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:21.527874 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:21.493853 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:25:21.527874 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:21.514205 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:25:21.617854 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:21.587921 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:25:21.851454 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:21.851418 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 04 01:25:21.852318 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:21.851913 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:25:21.893817 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:21.878743 4852 generic.go:356] "Generic (PLEG): container finished" podID="a84836f4-c505-422f-9adc-d96adcb96ced" containerID="5451bf1336b132e54e8d2b5a1e2d822c6f85bad1d952f4a106cb3a6439a23437" exitCode=0 Nov 04 01:25:21.893817 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:21.878951 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh" event={"ID":"a84836f4-c505-422f-9adc-d96adcb96ced","Type":"ContainerDied","Data":"5451bf1336b132e54e8d2b5a1e2d822c6f85bad1d952f4a106cb3a6439a23437"} Nov 04 01:25:21.985059 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:21.958161 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:21.985059 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:21.971304 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:25:22.031717 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.026246 4852 generic.go:356] "Generic (PLEG): container finished" podID="594f2fcb-8a9c-446e-bda1-7d320b1a7e6f" containerID="3e36731026f255d034b4814c3ae28b3dc6d6971a2e7d99dd6350fa3bc8487162" exitCode=0 Nov 04 01:25:22.031717 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.026360 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m" event={"ID":"594f2fcb-8a9c-446e-bda1-7d320b1a7e6f","Type":"ContainerDied","Data":"3e36731026f255d034b4814c3ae28b3dc6d6971a2e7d99dd6350fa3bc8487162"} Nov 04 01:25:22.093641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.058511 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2ttwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:25:22.093641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.058595 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" podUID="266b7c8e-ffa2-410b-a592-a96f3e034103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:22.093641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.060854 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": context deadline exceeded" Nov 04 01:25:22.153177 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.153053 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:22.154753 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.154732 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh" Nov 04 01:25:22.160300 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.157645 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m" Nov 04 01:25:22.163893 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.162803 4852 generic.go:356] "Generic (PLEG): container finished" podID="3c9e69da-ee72-4330-b006-baa850a853d9" containerID="40ca46b848f87ec684fa02149ee037adbafb1ba7da0c96b45ace7887f571ccc4" exitCode=0 Nov 04 01:25:22.213588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.212802 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:22.242591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.238969 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:22.316840 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.316806 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:22.376596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.347820 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a84836f4-c505-422f-9adc-d96adcb96ced-kube-api-access-tmwb5\" (UniqueName: \"kubernetes.io/projected/a84836f4-c505-422f-9adc-d96adcb96ced-kube-api-access-tmwb5\") pod \"a84836f4-c505-422f-9adc-d96adcb96ced\" (UID: \"a84836f4-c505-422f-9adc-d96adcb96ced\") " Nov 04 01:25:22.376596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.348021 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/594f2fcb-8a9c-446e-bda1-7d320b1a7e6f-kube-api-access-5glr2\" (UniqueName: \"kubernetes.io/projected/594f2fcb-8a9c-446e-bda1-7d320b1a7e6f-kube-api-access-5glr2\") pod \"594f2fcb-8a9c-446e-bda1-7d320b1a7e6f\" (UID: \"594f2fcb-8a9c-446e-bda1-7d320b1a7e6f\") " Nov 04 01:25:22.376596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.348095 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/594f2fcb-8a9c-446e-bda1-7d320b1a7e6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594f2fcb-8a9c-446e-bda1-7d320b1a7e6f-plugins-config-volume\") pod \"594f2fcb-8a9c-446e-bda1-7d320b1a7e6f\" (UID: \"594f2fcb-8a9c-446e-bda1-7d320b1a7e6f\") " Nov 04 01:25:22.376596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.348226 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a84836f4-c505-422f-9adc-d96adcb96ced-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a84836f4-c505-422f-9adc-d96adcb96ced-plugins-config-volume\") pod \"a84836f4-c505-422f-9adc-d96adcb96ced\" (UID: \"a84836f4-c505-422f-9adc-d96adcb96ced\") " Nov 04 01:25:22.376596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.350854 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:22.376596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.375784 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a84836f4-c505-422f-9adc-d96adcb96ced-plugins-config-volume" pod "a84836f4-c505-422f-9adc-d96adcb96ced" (UID: "a84836f4-c505-422f-9adc-d96adcb96ced"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:22.385803 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.385770 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/594f2fcb-8a9c-446e-bda1-7d320b1a7e6f-plugins-config-volume" pod "594f2fcb-8a9c-446e-bda1-7d320b1a7e6f" (UID: "594f2fcb-8a9c-446e-bda1-7d320b1a7e6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:22.482737 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.482707 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594f2fcb-8a9c-446e-bda1-7d320b1a7e6f-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:22.483484 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.483466 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a84836f4-c505-422f-9adc-d96adcb96ced-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:22.483802 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.483428 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6" event={"ID":"3c9e69da-ee72-4330-b006-baa850a853d9","Type":"ContainerDied","Data":"40ca46b848f87ec684fa02149ee037adbafb1ba7da0c96b45ace7887f571ccc4"} Nov 04 01:25:22.484927 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.484904 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a84836f4-c505-422f-9adc-d96adcb96ced-kube-api-access-tmwb5" pod "a84836f4-c505-422f-9adc-d96adcb96ced" (UID: "a84836f4-c505-422f-9adc-d96adcb96ced"). InnerVolumeSpecName "kube-api-access-tmwb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:22.497668 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.497635 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:22.504042 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.504018 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594f2fcb-8a9c-446e-bda1-7d320b1a7e6f-kube-api-access-5glr2" pod "594f2fcb-8a9c-446e-bda1-7d320b1a7e6f" (UID: "594f2fcb-8a9c-446e-bda1-7d320b1a7e6f"). InnerVolumeSpecName "kube-api-access-5glr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:22.524079 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.524049 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:22.524737 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.524717 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:25:22.600244 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.600200 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:22.611787 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.611757 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5glr2\" (UniqueName: \"kubernetes.io/projected/594f2fcb-8a9c-446e-bda1-7d320b1a7e6f-kube-api-access-5glr2\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:22.799591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.789625 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmwb5\" (UniqueName: \"kubernetes.io/projected/a84836f4-c505-422f-9adc-d96adcb96ced-kube-api-access-tmwb5\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:22.799591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.762843 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:22.867960 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.845731 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": context deadline exceeded" Nov 04 01:25:22.867960 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.861538 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:22.900842 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:22.883694 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:23.050640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:23.049556 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:23.141586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:23.092297 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk"] Nov 04 01:25:23.141586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:23.128672 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:23.141586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:23.129260 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" podUID="78a2fbe6-6fca-428a-9bf6-43bb549282ff" containerName="maas-api" containerID="cri-o://e1066bb77ed1b6fb4def9d623bda2bf8eb049e70d2a70c42b05242c1ffe43adf" gracePeriod=30 Nov 04 01:25:23.195635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:23.190739 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:23.276264 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:23.276196 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:23.276848 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:23.276818 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:23.279217 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:23.279196 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:25:23.290593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:23.285528 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:25:23.298041 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:23.297996 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:25:23.355624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:23.303747 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:23.356089 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:23.356043 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:23.575832 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:23.535694 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:25:23.831048 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:23.827703 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 04 01:25:23.847850 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:23.847192 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:25:23.926784 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:23.908487 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:24.035588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.004711 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 04 01:25:24.035588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.025900 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:24.059591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.044880 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" podUID="78a2fbe6-6fca-428a-9bf6-43bb549282ff" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.73:8443/health\": dial tcp 10.134.0.73:8443: connect: connection refused" Nov 04 01:25:24.059591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.053768 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:24.063003 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.062979 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6" Nov 04 01:25:24.120239 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.120142 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:24.227107 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.203773 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:24.228599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.223126 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:24.309590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.299660 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:24.321588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.316278 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh" Nov 04 01:25:24.335591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.327869 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m" Nov 04 01:25:24.342847 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.342407 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6" Nov 04 01:25:24.344606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.344374 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:24.350713 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.350678 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:24.358184 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.358152 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:24.397451 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.397380 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:25:24.397713 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.397695 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:25:24.397896 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.397883 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt"] Nov 04 01:25:24.398057 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.398046 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:25:24.398244 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.398223 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh" event={"ID":"a84836f4-c505-422f-9adc-d96adcb96ced","Type":"ContainerDied","Data":"25657dc0c7890d7a96deaf7c956badb016089323eb5b07f639d9f92bca795f7e"} Nov 04 01:25:24.398373 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.398355 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m" event={"ID":"594f2fcb-8a9c-446e-bda1-7d320b1a7e6f","Type":"ContainerDied","Data":"229cbc375435ef81133dbfc2b2caeb389b6a1bee1bf1bdc32c7d5781fbc28929"} Nov 04 01:25:24.398476 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.398461 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6" event={"ID":"3c9e69da-ee72-4330-b006-baa850a853d9","Type":"ContainerDied","Data":"a3f202fa1ba6802bcff42c559594737dbea02aa22d7c69e1d365066958c11e72"} Nov 04 01:25:24.398651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.398636 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:25:24.398776 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.398765 4852 scope.go:122] "RemoveContainer" containerID="5451bf1336b132e54e8d2b5a1e2d822c6f85bad1d952f4a106cb3a6439a23437" Nov 04 01:25:24.434501 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.432281 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c9e69da-ee72-4330-b006-baa850a853d9-kube-api-access-fchvv\" (UniqueName: \"kubernetes.io/projected/3c9e69da-ee72-4330-b006-baa850a853d9-kube-api-access-fchvv\") pod \"3c9e69da-ee72-4330-b006-baa850a853d9\" (UID: \"3c9e69da-ee72-4330-b006-baa850a853d9\") " Nov 04 01:25:24.434501 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.432403 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c9e69da-ee72-4330-b006-baa850a853d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9e69da-ee72-4330-b006-baa850a853d9-plugins-config-volume\") pod \"3c9e69da-ee72-4330-b006-baa850a853d9\" (UID: \"3c9e69da-ee72-4330-b006-baa850a853d9\") " Nov 04 01:25:24.439601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.436772 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c9e69da-ee72-4330-b006-baa850a853d9-plugins-config-volume" pod "3c9e69da-ee72-4330-b006-baa850a853d9" (UID: "3c9e69da-ee72-4330-b006-baa850a853d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:24.519979 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.519316 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9e69da-ee72-4330-b006-baa850a853d9-kube-api-access-fchvv" pod "3c9e69da-ee72-4330-b006-baa850a853d9" (UID: "3c9e69da-ee72-4330-b006-baa850a853d9"). InnerVolumeSpecName "kube-api-access-fchvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:24.536510 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.536480 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9e69da-ee72-4330-b006-baa850a853d9-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:24.536743 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.536728 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fchvv\" (UniqueName: \"kubernetes.io/projected/3c9e69da-ee72-4330-b006-baa850a853d9-kube-api-access-fchvv\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:24.544235 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.544147 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" podUID="d84493fd-2664-4f56-8542-84072c8850e2" containerName="istio-proxy" containerID="cri-o://8887f68abf5e204265e5753c043f656e3dea5adf66b1c098186549a8449239d8" gracePeriod=30 Nov 04 01:25:24.544389 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.544355 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:24.661759 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.650462 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:25:24.661759 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.651719 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:24.740238 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.739852 4852 scope.go:122] "RemoveContainer" containerID="3e36731026f255d034b4814c3ae28b3dc6d6971a2e7d99dd6350fa3bc8487162" Nov 04 01:25:24.750814 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:24.750688 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:25:25.068639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.025397 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 04 01:25:25.068639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.041528 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:25.068639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.063905 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:25:25.158623 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.074763 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:25.250148 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.087209 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 04 01:25:25.294110 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.294081 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:25:25.306274 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.306234 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:25.328291 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.328262 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:25:25.356678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.340702 4852 scope.go:122] "RemoveContainer" containerID="40ca46b848f87ec684fa02149ee037adbafb1ba7da0c96b45ace7887f571ccc4" Nov 04 01:25:25.374073 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.374019 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:25.520352 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.478075 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:25.559640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.554091 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:25:25.570335 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.570026 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:25:25.603826 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.603795 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:25.684232 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.684202 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m"] Nov 04 01:25:25.684416 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.684313 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:25:25.684416 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.684383 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:25:25.699516 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.699494 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:25:25.752934 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.744249 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d92c65cb-59cf49df46-kgj9m"] Nov 04 01:25:25.753264 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.753221 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" event={"ID":"78a2fbe6-6fca-428a-9bf6-43bb549282ff","Type":"ContainerDied","Data":"e1066bb77ed1b6fb4def9d623bda2bf8eb049e70d2a70c42b05242c1ffe43adf"} Nov 04 01:25:25.753430 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.749740 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" Nov 04 01:25:25.753547 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.745769 4852 generic.go:356] "Generic (PLEG): container finished" podID="78a2fbe6-6fca-428a-9bf6-43bb549282ff" containerID="e1066bb77ed1b6fb4def9d623bda2bf8eb049e70d2a70c42b05242c1ffe43adf" exitCode=0 Nov 04 01:25:25.794282 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.794232 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh"] Nov 04 01:25:25.794552 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.794527 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d92c65cb-7cbffb4ddc-xf6mh"] Nov 04 01:25:25.818389 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.818337 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:25.858870 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.858785 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78a2fbe6-6fca-428a-9bf6-43bb549282ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a2fbe6-6fca-428a-9bf6-43bb549282ff-openshift-service-ca\") pod \"78a2fbe6-6fca-428a-9bf6-43bb549282ff\" (UID: \"78a2fbe6-6fca-428a-9bf6-43bb549282ff\") " Nov 04 01:25:25.859197 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.859173 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78a2fbe6-6fca-428a-9bf6-43bb549282ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78a2fbe6-6fca-428a-9bf6-43bb549282ff-maas-api-tls\") pod \"78a2fbe6-6fca-428a-9bf6-43bb549282ff\" (UID: \"78a2fbe6-6fca-428a-9bf6-43bb549282ff\") " Nov 04 01:25:25.859413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.859392 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78a2fbe6-6fca-428a-9bf6-43bb549282ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/78a2fbe6-6fca-428a-9bf6-43bb549282ff-metrics-certs\") pod \"78a2fbe6-6fca-428a-9bf6-43bb549282ff\" (UID: \"78a2fbe6-6fca-428a-9bf6-43bb549282ff\") " Nov 04 01:25:25.859635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.859610 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78a2fbe6-6fca-428a-9bf6-43bb549282ff-kube-api-access-twfnh\" (UniqueName: \"kubernetes.io/projected/78a2fbe6-6fca-428a-9bf6-43bb549282ff-kube-api-access-twfnh\") pod \"78a2fbe6-6fca-428a-9bf6-43bb549282ff\" (UID: \"78a2fbe6-6fca-428a-9bf6-43bb549282ff\") " Nov 04 01:25:25.861403 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.861378 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78a2fbe6-6fca-428a-9bf6-43bb549282ff-openshift-service-ca" pod "78a2fbe6-6fca-428a-9bf6-43bb549282ff" (UID: "78a2fbe6-6fca-428a-9bf6-43bb549282ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:25.939843 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:25.939800 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78a2fbe6-6fca-428a-9bf6-43bb549282ff-maas-api-tls" pod "78a2fbe6-6fca-428a-9bf6-43bb549282ff" (UID: "78a2fbe6-6fca-428a-9bf6-43bb549282ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:25:26.131639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:26.086005 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a2fbe6-6fca-428a-9bf6-43bb549282ff-kube-api-access-twfnh" pod "78a2fbe6-6fca-428a-9bf6-43bb549282ff" (UID: "78a2fbe6-6fca-428a-9bf6-43bb549282ff"). InnerVolumeSpecName "kube-api-access-twfnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:26.167120 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:26.167079 4852 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a2fbe6-6fca-428a-9bf6-43bb549282ff-openshift-service-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:26.178454 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:26.178421 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78a2fbe6-6fca-428a-9bf6-43bb549282ff-maas-api-tls\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:26.186087 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:26.178665 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twfnh\" (UniqueName: \"kubernetes.io/projected/78a2fbe6-6fca-428a-9bf6-43bb549282ff-kube-api-access-twfnh\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:26.261893 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:26.261858 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:26.339960 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:26.339921 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78a2fbe6-6fca-428a-9bf6-43bb549282ff-metrics-certs" pod "78a2fbe6-6fca-428a-9bf6-43bb549282ff" (UID: "78a2fbe6-6fca-428a-9bf6-43bb549282ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:25:26.453376 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:26.453288 4852 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78a2fbe6-6fca-428a-9bf6-43bb549282ff-metrics-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:26.537912 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:26.537877 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="594f2fcb-8a9c-446e-bda1-7d320b1a7e6f" path="/var/lib/kubelet/pods/594f2fcb-8a9c-446e-bda1-7d320b1a7e6f/volumes" Nov 04 01:25:26.538850 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:26.538828 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a84836f4-c505-422f-9adc-d96adcb96ced" path="/var/lib/kubelet/pods/a84836f4-c505-422f-9adc-d96adcb96ced/volumes" Nov 04 01:25:26.555588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:26.549133 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6"] Nov 04 01:25:26.555588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:26.549178 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7b10de73-ddf6c964c-hh8l6"] Nov 04 01:25:26.823326 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:26.814737 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:26.881865 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:26.881830 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" event={"ID":"78a2fbe6-6fca-428a-9bf6-43bb549282ff","Type":"ContainerDied","Data":"48e47237996a31923cf164487d2f0c9c96a2de7adfcb9bddc7e33b0f832d99b3"} Nov 04 01:25:26.882118 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:26.882103 4852 scope.go:122] "RemoveContainer" containerID="e1066bb77ed1b6fb4def9d623bda2bf8eb049e70d2a70c42b05242c1ffe43adf" Nov 04 01:25:26.882323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:26.882306 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk" Nov 04 01:25:26.918538 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:26.918516 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk"] Nov 04 01:25:26.922883 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:26.922850 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d92c65cb-b7956b695-zzhpk"] Nov 04 01:25:27.549477 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:27.549433 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:27.803253 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:27.803170 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:27.997160 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:27.997123 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c9e69da-ee72-4330-b006-baa850a853d9" path="/var/lib/kubelet/pods/3c9e69da-ee72-4330-b006-baa850a853d9/volumes" Nov 04 01:25:27.997823 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:27.997801 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78a2fbe6-6fca-428a-9bf6-43bb549282ff" path="/var/lib/kubelet/pods/78a2fbe6-6fca-428a-9bf6-43bb549282ff/volumes" Nov 04 01:25:28.406647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:28.405187 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:29.141802 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:29.141771 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:29.868403 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:25:29.868354 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:25:30.005108 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:30.004983 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:31.028245 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:31.028200 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:31.764639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:31.764555 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:32.281648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.281621 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:25:32.401910 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.401825 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-workload-socket\") pod \"d84493fd-2664-4f56-8542-84072c8850e2\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " Nov 04 01:25:32.402192 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.402168 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-workload-certs\") pod \"d84493fd-2664-4f56-8542-84072c8850e2\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " Nov 04 01:25:32.402373 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.402356 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d84493fd-2664-4f56-8542-84072c8850e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84493fd-2664-4f56-8542-84072c8850e2-istio-podinfo\") pod \"d84493fd-2664-4f56-8542-84072c8850e2\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " Nov 04 01:25:32.402511 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.402494 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d84493fd-2664-4f56-8542-84072c8850e2-kube-api-access-dpp5p\" (UniqueName: \"kubernetes.io/projected/d84493fd-2664-4f56-8542-84072c8850e2-kube-api-access-dpp5p\") pod \"d84493fd-2664-4f56-8542-84072c8850e2\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " Nov 04 01:25:32.402701 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.402683 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d84493fd-2664-4f56-8542-84072c8850e2-istio-token\" (UniqueName: \"kubernetes.io/projected/d84493fd-2664-4f56-8542-84072c8850e2-istio-token\") pod \"d84493fd-2664-4f56-8542-84072c8850e2\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " Nov 04 01:25:32.402918 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.402893 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-istio-envoy\") pod \"d84493fd-2664-4f56-8542-84072c8850e2\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " Nov 04 01:25:32.403087 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.402971 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d84493fd-2664-4f56-8542-84072c8850e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84493fd-2664-4f56-8542-84072c8850e2-istiod-ca-cert\") pod \"d84493fd-2664-4f56-8542-84072c8850e2\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " Nov 04 01:25:32.403087 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.403051 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-credential-socket\") pod \"d84493fd-2664-4f56-8542-84072c8850e2\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " Nov 04 01:25:32.411380 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.403187 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-istio-data\") pod \"d84493fd-2664-4f56-8542-84072c8850e2\" (UID: \"d84493fd-2664-4f56-8542-84072c8850e2\") " Nov 04 01:25:32.411380 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.404135 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-istio-data" pod "d84493fd-2664-4f56-8542-84072c8850e2" (UID: "d84493fd-2664-4f56-8542-84072c8850e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:32.411380 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.404326 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-workload-socket" pod "d84493fd-2664-4f56-8542-84072c8850e2" (UID: "d84493fd-2664-4f56-8542-84072c8850e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:32.411380 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.404633 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-workload-certs" pod "d84493fd-2664-4f56-8542-84072c8850e2" (UID: "d84493fd-2664-4f56-8542-84072c8850e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:32.411380 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.405150 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d84493fd-2664-4f56-8542-84072c8850e2-istiod-ca-cert" pod "d84493fd-2664-4f56-8542-84072c8850e2" (UID: "d84493fd-2664-4f56-8542-84072c8850e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:32.411380 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.405831 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-credential-socket" pod "d84493fd-2664-4f56-8542-84072c8850e2" (UID: "d84493fd-2664-4f56-8542-84072c8850e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:32.411380 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.408475 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-istio-envoy" pod "d84493fd-2664-4f56-8542-84072c8850e2" (UID: "d84493fd-2664-4f56-8542-84072c8850e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:32.411380 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.410963 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d84493fd-2664-4f56-8542-84072c8850e2-istio-podinfo" pod "d84493fd-2664-4f56-8542-84072c8850e2" (UID: "d84493fd-2664-4f56-8542-84072c8850e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:25:32.411866 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.411389 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d84493fd-2664-4f56-8542-84072c8850e2-istio-token" pod "d84493fd-2664-4f56-8542-84072c8850e2" (UID: "d84493fd-2664-4f56-8542-84072c8850e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:32.411922 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.411893 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d84493fd-2664-4f56-8542-84072c8850e2-kube-api-access-dpp5p" pod "d84493fd-2664-4f56-8542-84072c8850e2" (UID: "d84493fd-2664-4f56-8542-84072c8850e2"). InnerVolumeSpecName "kube-api-access-dpp5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:32.441349 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.438586 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:32.517954 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.517795 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84493fd-2664-4f56-8542-84072c8850e2-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:32.517954 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.517831 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:32.517954 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.517847 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:32.517954 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.517860 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:32.517954 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.517873 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:32.517954 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.517888 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84493fd-2664-4f56-8542-84072c8850e2-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:32.517954 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.517901 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpp5p\" (UniqueName: \"kubernetes.io/projected/d84493fd-2664-4f56-8542-84072c8850e2-kube-api-access-dpp5p\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:32.517954 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.517916 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84493fd-2664-4f56-8542-84072c8850e2-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:32.517954 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.517929 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84493fd-2664-4f56-8542-84072c8850e2-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:32.852510 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.852474 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:32.986659 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.986622 4852 generic.go:356] "Generic (PLEG): container finished" podID="d84493fd-2664-4f56-8542-84072c8850e2" containerID="8887f68abf5e204265e5753c043f656e3dea5adf66b1c098186549a8449239d8" exitCode=0 Nov 04 01:25:32.986850 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.986658 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" event={"ID":"d84493fd-2664-4f56-8542-84072c8850e2","Type":"ContainerDied","Data":"8887f68abf5e204265e5753c043f656e3dea5adf66b1c098186549a8449239d8"} Nov 04 01:25:32.986850 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.986678 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" Nov 04 01:25:32.986850 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.986722 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt" event={"ID":"d84493fd-2664-4f56-8542-84072c8850e2","Type":"ContainerDied","Data":"5559104f022dd164974d3374e536540d95c13e56914554ba6c16e6132c12dd47"} Nov 04 01:25:32.986850 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.986753 4852 scope.go:122] "RemoveContainer" containerID="8887f68abf5e204265e5753c043f656e3dea5adf66b1c098186549a8449239d8" Nov 04 01:25:32.997682 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.997666 4852 scope.go:122] "RemoveContainer" containerID="8887f68abf5e204265e5753c043f656e3dea5adf66b1c098186549a8449239d8" Nov 04 01:25:32.997934 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:25:32.997906 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8887f68abf5e204265e5753c043f656e3dea5adf66b1c098186549a8449239d8\": container with ID starting with 8887f68abf5e204265e5753c043f656e3dea5adf66b1c098186549a8449239d8 not found: ID does not exist" containerID="8887f68abf5e204265e5753c043f656e3dea5adf66b1c098186549a8449239d8" Nov 04 01:25:32.997999 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:32.997935 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8887f68abf5e204265e5753c043f656e3dea5adf66b1c098186549a8449239d8"} err="failed to get container status \"8887f68abf5e204265e5753c043f656e3dea5adf66b1c098186549a8449239d8\": rpc error: code = NotFound desc = could not find container \"8887f68abf5e204265e5753c043f656e3dea5adf66b1c098186549a8449239d8\": container with ID starting with 8887f68abf5e204265e5753c043f656e3dea5adf66b1c098186549a8449239d8 not found: ID does not exist" Nov 04 01:25:33.012890 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:33.012868 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt"] Nov 04 01:25:33.018965 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:33.018944 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d92c65cb-openshift-default-fc7654589-vs5wt"] Nov 04 01:25:33.519652 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:33.517137 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:33.682928 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:33.682894 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:33.995253 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:33.995179 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d84493fd-2664-4f56-8542-84072c8850e2" path="/var/lib/kubelet/pods/d84493fd-2664-4f56-8542-84072c8850e2/volumes" Nov 04 01:25:34.280590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:34.278113 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:35.127172 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.127143 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:35.252599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.251981 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5"] Nov 04 01:25:35.254634 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.253161 4852 cpu_manager.go:414] "containerMap: removing container" podUID="3c9e69da-ee72-4330-b006-baa850a853d9" containerName="payload-pre-processing" Nov 04 01:25:35.254634 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.253184 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c9e69da-ee72-4330-b006-baa850a853d9" containerName="payload-pre-processing" Nov 04 01:25:35.260639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.258623 4852 cpu_manager.go:414] "containerMap: removing container" podUID="594f2fcb-8a9c-446e-bda1-7d320b1a7e6f" containerName="payload-processing" Nov 04 01:25:35.260639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.258649 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="594f2fcb-8a9c-446e-bda1-7d320b1a7e6f" containerName="payload-processing" Nov 04 01:25:35.260639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.258756 4852 cpu_manager.go:414] "containerMap: removing container" podUID="a84836f4-c505-422f-9adc-d96adcb96ced" containerName="payload-pre-processing" Nov 04 01:25:35.260639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.258765 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a84836f4-c505-422f-9adc-d96adcb96ced" containerName="payload-pre-processing" Nov 04 01:25:35.260639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.258779 4852 cpu_manager.go:414] "containerMap: removing container" podUID="78a2fbe6-6fca-428a-9bf6-43bb549282ff" containerName="maas-api" Nov 04 01:25:35.260639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.258787 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78a2fbe6-6fca-428a-9bf6-43bb549282ff" containerName="maas-api" Nov 04 01:25:35.260639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.258835 4852 cpu_manager.go:414] "containerMap: removing container" podUID="d84493fd-2664-4f56-8542-84072c8850e2" containerName="istio-proxy" Nov 04 01:25:35.260639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.258843 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d84493fd-2664-4f56-8542-84072c8850e2" containerName="istio-proxy" Nov 04 01:25:35.260639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.259045 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="78a2fbe6-6fca-428a-9bf6-43bb549282ff" containerName="maas-api" Nov 04 01:25:35.260639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.259079 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="d84493fd-2664-4f56-8542-84072c8850e2" containerName="istio-proxy" Nov 04 01:25:35.260639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.259092 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c9e69da-ee72-4330-b006-baa850a853d9" containerName="payload-pre-processing" Nov 04 01:25:35.260639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.259104 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="594f2fcb-8a9c-446e-bda1-7d320b1a7e6f" containerName="payload-processing" Nov 04 01:25:35.260639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.259119 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="a84836f4-c505-422f-9adc-d96adcb96ced" containerName="payload-pre-processing" Nov 04 01:25:35.291057 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.291027 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.298208 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.298175 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5"] Nov 04 01:25:35.374502 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.374383 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d7024f0c-openshift-default-dockercfg-bzddd\"" Nov 04 01:25:35.379378 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.378313 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-podinfo\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.379378 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.378379 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-workload-socket\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.379378 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.378433 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqlf5\" (UniqueName: \"kubernetes.io/projected/eef87d33-f22e-4ce2-ad38-94ac8f23913f-kube-api-access-mqlf5\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.379378 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.378539 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istiod-ca-cert\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.379378 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.378689 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-token\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.379378 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.378788 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-envoy\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.379378 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.378850 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-data\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.379378 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.378883 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-workload-certs\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.379378 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.378937 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-credential-socket\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.480732 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.480690 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istiod-ca-cert\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.480926 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.480833 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-token\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.480999 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.480970 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-envoy\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.481083 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.481057 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-data\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.481140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.481100 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-workload-certs\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.481202 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.481152 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-credential-socket\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.482199 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.481306 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-podinfo\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.482199 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.481352 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-workload-socket\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.482199 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.481407 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqlf5\" (UniqueName: \"kubernetes.io/projected/eef87d33-f22e-4ce2-ad38-94ac8f23913f-kube-api-access-mqlf5\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.482199 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.481736 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istiod-ca-cert\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.482199 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.482003 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-workload-certs\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.482199 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.482036 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-workload-socket\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.482549 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.482272 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-credential-socket\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.482549 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.482308 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-data\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.484653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.484621 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-podinfo\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.484807 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.484787 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-envoy\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.489230 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.489208 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-token\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.489783 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.489766 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqlf5\" (UniqueName: \"kubernetes.io/projected/eef87d33-f22e-4ce2-ad38-94ac8f23913f-kube-api-access-mqlf5\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.700773 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.700698 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:35.741843 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.741813 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:35.963146 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:35.963067 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:36.001101 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:36.001073 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5"] Nov 04 01:25:36.005322 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:25:36.005296 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef87d33_f22e_4ce2_ad38_94ac8f23913f.slice/crio-e0c140e33677d6de643641e90ef6064a806fc27e812e24f17c98240aa911a9a7 WatchSource:0}: Error finding container e0c140e33677d6de643641e90ef6064a806fc27e812e24f17c98240aa911a9a7: Status 404 returned error can't find the container with id e0c140e33677d6de643641e90ef6064a806fc27e812e24f17c98240aa911a9a7 Nov 04 01:25:36.007870 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:36.007838 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:25:36.007954 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:36.007922 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:25:36.007998 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:36.007953 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:25:36.520053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:36.520004 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:36.698419 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:36.698384 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:37.014458 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.014428 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" event={"ID":"eef87d33-f22e-4ce2-ad38-94ac8f23913f","Type":"ContainerStarted","Data":"49e5c5d885267acc74698223540a0a17f037c410cee3ef230dc552061e0bdc96"} Nov 04 01:25:37.014671 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.014475 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" event={"ID":"eef87d33-f22e-4ce2-ad38-94ac8f23913f","Type":"ContainerStarted","Data":"e0c140e33677d6de643641e90ef6064a806fc27e812e24f17c98240aa911a9a7"} Nov 04 01:25:37.595821 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.592852 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:37.701725 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.701689 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:37.703140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.703112 4852 patch_prober.go:29] interesting pod/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:25:37.703266 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.703156 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" podUID="eef87d33-f22e-4ce2-ad38-94ac8f23913f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 04 01:25:37.722171 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.722126 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" podStartSLOduration=2.722111904 podStartE2EDuration="2.722111904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:37.046726731 +0000 UTC m=+1619.161836848" watchObservedRunningTime="2026-11-04 01:25:37.722111904 +0000 UTC m=+1619.837222022" Nov 04 01:25:37.724120 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.724103 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q"] Nov 04 01:25:37.728212 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.728194 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q" Nov 04 01:25:37.737905 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.737354 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q"] Nov 04 01:25:37.739144 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.739124 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-62510d3b-dockercfg-fnxbg\"" Nov 04 01:25:37.739318 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.739294 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-62510d3b\"" Nov 04 01:25:37.779908 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.779867 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb"] Nov 04 01:25:37.789654 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.789487 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:37.803643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.803614 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb" Nov 04 01:25:37.808652 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.808388 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb"] Nov 04 01:25:37.821601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.821550 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce052f4b-e92c-4795-a4ba-947cf1c824cf-plugins-config-volume\") pod \"payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q\" (UID: \"ce052f4b-e92c-4795-a4ba-947cf1c824cf\") " pod="openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q" Nov 04 01:25:37.821915 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.821895 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqrcg\" (UniqueName: \"kubernetes.io/projected/ce052f4b-e92c-4795-a4ba-947cf1c824cf-kube-api-access-xqrcg\") pod \"payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q\" (UID: \"ce052f4b-e92c-4795-a4ba-947cf1c824cf\") " pod="openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q" Nov 04 01:25:37.935637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.934069 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqrcg\" (UniqueName: \"kubernetes.io/projected/ce052f4b-e92c-4795-a4ba-947cf1c824cf-kube-api-access-xqrcg\") pod \"payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q\" (UID: \"ce052f4b-e92c-4795-a4ba-947cf1c824cf\") " pod="openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q" Nov 04 01:25:37.935637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.934133 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea5f3c5-a60c-4781-87de-89edbb87c02a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb\" (UID: \"3ea5f3c5-a60c-4781-87de-89edbb87c02a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb" Nov 04 01:25:37.935637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.934206 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce052f4b-e92c-4795-a4ba-947cf1c824cf-plugins-config-volume\") pod \"payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q\" (UID: \"ce052f4b-e92c-4795-a4ba-947cf1c824cf\") " pod="openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q" Nov 04 01:25:37.935637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.934318 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-779tg\" (UniqueName: \"kubernetes.io/projected/3ea5f3c5-a60c-4781-87de-89edbb87c02a-kube-api-access-779tg\") pod \"payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb\" (UID: \"3ea5f3c5-a60c-4781-87de-89edbb87c02a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb" Nov 04 01:25:37.993257 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.993212 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:37.993532 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:37.993511 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce052f4b-e92c-4795-a4ba-947cf1c824cf-plugins-config-volume\") pod \"payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q\" (UID: \"ce052f4b-e92c-4795-a4ba-947cf1c824cf\") " pod="openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q" Nov 04 01:25:38.036248 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:38.030799 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqrcg\" (UniqueName: \"kubernetes.io/projected/ce052f4b-e92c-4795-a4ba-947cf1c824cf-kube-api-access-xqrcg\") pod \"payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q\" (UID: \"ce052f4b-e92c-4795-a4ba-947cf1c824cf\") " pod="openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q" Nov 04 01:25:38.037840 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:38.037805 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea5f3c5-a60c-4781-87de-89edbb87c02a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb\" (UID: \"3ea5f3c5-a60c-4781-87de-89edbb87c02a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb" Nov 04 01:25:38.038222 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:38.038193 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-779tg\" (UniqueName: \"kubernetes.io/projected/3ea5f3c5-a60c-4781-87de-89edbb87c02a-kube-api-access-779tg\") pod \"payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb\" (UID: \"3ea5f3c5-a60c-4781-87de-89edbb87c02a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb" Nov 04 01:25:38.042373 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:38.042339 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea5f3c5-a60c-4781-87de-89edbb87c02a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb\" (UID: \"3ea5f3c5-a60c-4781-87de-89edbb87c02a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb" Nov 04 01:25:38.090668 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:38.090635 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-62510d3b-dockercfg-fnxbg\"" Nov 04 01:25:38.094660 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:38.094619 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q" Nov 04 01:25:38.206640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:38.198215 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-779tg\" (UniqueName: \"kubernetes.io/projected/3ea5f3c5-a60c-4781-87de-89edbb87c02a-kube-api-access-779tg\") pod \"payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb\" (UID: \"3ea5f3c5-a60c-4781-87de-89edbb87c02a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb" Nov 04 01:25:38.422112 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:38.422081 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb" Nov 04 01:25:38.529985 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:38.529957 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q"] Nov 04 01:25:38.705590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:38.702420 4852 patch_prober.go:29] interesting pod/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:25:38.705590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:38.702477 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" podUID="eef87d33-f22e-4ce2-ad38-94ac8f23913f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 04 01:25:38.811238 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:38.781578 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:38.950912 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:38.950881 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:39.250160 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.234813 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf"] Nov 04 01:25:39.380653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.369890 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb"] Nov 04 01:25:39.380653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.369941 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf"] Nov 04 01:25:39.380653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.369967 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2"] Nov 04 01:25:39.380653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.373938 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf" Nov 04 01:25:39.381039 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.380839 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-296c124f\"" Nov 04 01:25:39.450032 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.449765 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2"] Nov 04 01:25:39.450032 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.449872 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2" Nov 04 01:25:39.455956 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.455898 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-296c124f-dockercfg-7ll9s\"" Nov 04 01:25:39.460598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.458355 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb" event={"ID":"3ea5f3c5-a60c-4781-87de-89edbb87c02a","Type":"ContainerStarted","Data":"0ba87f9bbd8dcf41924c4ba341798563db1e13a9279a7f8683a50c97da2b240e"} Nov 04 01:25:39.460598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.458991 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc338560-6a2d-4fdd-bd5c-23e1071d63a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf\" (UID: \"dc338560-6a2d-4fdd-bd5c-23e1071d63a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf" Nov 04 01:25:39.460598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.459134 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv2rl\" (UniqueName: \"kubernetes.io/projected/dc338560-6a2d-4fdd-bd5c-23e1071d63a7-kube-api-access-vv2rl\") pod \"payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf\" (UID: \"dc338560-6a2d-4fdd-bd5c-23e1071d63a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf" Nov 04 01:25:39.500494 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.500440 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q" event={"ID":"ce052f4b-e92c-4795-a4ba-947cf1c824cf","Type":"ContainerStarted","Data":"0c500289540304592268f997aefc1b370997e6dca7877eac9cfd060fbfce50c3"} Nov 04 01:25:39.568625 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.568555 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv2rl\" (UniqueName: \"kubernetes.io/projected/dc338560-6a2d-4fdd-bd5c-23e1071d63a7-kube-api-access-vv2rl\") pod \"payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf\" (UID: \"dc338560-6a2d-4fdd-bd5c-23e1071d63a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf" Nov 04 01:25:39.568810 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.568707 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg8xs\" (UniqueName: \"kubernetes.io/projected/514b08d9-34ac-4867-90a4-53fd518d25c1-kube-api-access-mg8xs\") pod \"payload-processing-e2e-ait-296c124f-7bb687db96-25vb2\" (UID: \"514b08d9-34ac-4867-90a4-53fd518d25c1\") " pod="openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2" Nov 04 01:25:39.568884 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.568851 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514b08d9-34ac-4867-90a4-53fd518d25c1-plugins-config-volume\") pod \"payload-processing-e2e-ait-296c124f-7bb687db96-25vb2\" (UID: \"514b08d9-34ac-4867-90a4-53fd518d25c1\") " pod="openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2" Nov 04 01:25:39.568943 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.568932 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc338560-6a2d-4fdd-bd5c-23e1071d63a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf\" (UID: \"dc338560-6a2d-4fdd-bd5c-23e1071d63a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf" Nov 04 01:25:39.573613 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.569799 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc338560-6a2d-4fdd-bd5c-23e1071d63a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf\" (UID: \"dc338560-6a2d-4fdd-bd5c-23e1071d63a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf" Nov 04 01:25:39.589382 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.589344 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv2rl\" (UniqueName: \"kubernetes.io/projected/dc338560-6a2d-4fdd-bd5c-23e1071d63a7-kube-api-access-vv2rl\") pod \"payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf\" (UID: \"dc338560-6a2d-4fdd-bd5c-23e1071d63a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf" Nov 04 01:25:39.682643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.682202 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg8xs\" (UniqueName: \"kubernetes.io/projected/514b08d9-34ac-4867-90a4-53fd518d25c1-kube-api-access-mg8xs\") pod \"payload-processing-e2e-ait-296c124f-7bb687db96-25vb2\" (UID: \"514b08d9-34ac-4867-90a4-53fd518d25c1\") " pod="openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2" Nov 04 01:25:39.682643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.682412 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514b08d9-34ac-4867-90a4-53fd518d25c1-plugins-config-volume\") pod \"payload-processing-e2e-ait-296c124f-7bb687db96-25vb2\" (UID: \"514b08d9-34ac-4867-90a4-53fd518d25c1\") " pod="openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2" Nov 04 01:25:39.683591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.683317 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514b08d9-34ac-4867-90a4-53fd518d25c1-plugins-config-volume\") pod \"payload-processing-e2e-ait-296c124f-7bb687db96-25vb2\" (UID: \"514b08d9-34ac-4867-90a4-53fd518d25c1\") " pod="openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2" Nov 04 01:25:39.695899 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.695875 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg8xs\" (UniqueName: \"kubernetes.io/projected/514b08d9-34ac-4867-90a4-53fd518d25c1-kube-api-access-mg8xs\") pod \"payload-processing-e2e-ait-296c124f-7bb687db96-25vb2\" (UID: \"514b08d9-34ac-4867-90a4-53fd518d25c1\") " pod="openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2" Nov 04 01:25:39.702681 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.702657 4852 patch_prober.go:29] interesting pod/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:25:39.702803 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.702698 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" podUID="eef87d33-f22e-4ce2-ad38-94ac8f23913f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 04 01:25:39.771669 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.764215 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:39.786590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.784101 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf" Nov 04 01:25:39.849153 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.849092 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2" Nov 04 01:25:39.906443 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.906421 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd"] Nov 04 01:25:39.929597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:39.929537 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5"] Nov 04 01:25:40.048815 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:40.048790 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:40.247468 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:40.247390 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:40.423589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:40.405629 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf"] Nov 04 01:25:40.500590 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:25:40.500276 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc338560_6a2d_4fdd_bd5c_23e1071d63a7.slice/crio-ca549dd19edbcbddf3366948189e541d3f8751091b37a84dfa20a08923badacb WatchSource:0}: Error finding container ca549dd19edbcbddf3366948189e541d3f8751091b37a84dfa20a08923badacb: Status 404 returned error can't find the container with id ca549dd19edbcbddf3366948189e541d3f8751091b37a84dfa20a08923badacb Nov 04 01:25:40.545114 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:40.545081 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2"] Nov 04 01:25:40.564076 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:25:40.564048 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514b08d9_34ac_4867_90a4_53fd518d25c1.slice/crio-cb0d012b21594d19e37ab88fa30c7184ca88857a5e3180e216c06c605e5265d5 WatchSource:0}: Error finding container cb0d012b21594d19e37ab88fa30c7184ca88857a5e3180e216c06c605e5265d5: Status 404 returned error can't find the container with id cb0d012b21594d19e37ab88fa30c7184ca88857a5e3180e216c06c605e5265d5 Nov 04 01:25:40.615816 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:40.615780 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb" event={"ID":"3ea5f3c5-a60c-4781-87de-89edbb87c02a","Type":"ContainerStarted","Data":"177f3d8608fac8cd5e5d126369a47adfbe50ba73bba9404ae9093e73b992476e"} Nov 04 01:25:40.615932 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:40.615907 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb" Nov 04 01:25:40.618836 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:40.618811 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q" event={"ID":"ce052f4b-e92c-4795-a4ba-947cf1c824cf","Type":"ContainerStarted","Data":"1b11c9560f937fe974a749feab5f9a98e795c9e9b07b8feab6eac0ae31c1afd0"} Nov 04 01:25:40.619368 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:40.619351 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q" Nov 04 01:25:40.639833 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:40.639791 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb" podStartSLOduration=3.63977798 podStartE2EDuration="3.63977798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:40.636276484 +0000 UTC m=+1622.751386602" watchObservedRunningTime="2026-11-04 01:25:40.63977798 +0000 UTC m=+1622.754888097" Nov 04 01:25:40.654852 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:40.654798 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q" podStartSLOduration=3.654784441 podStartE2EDuration="3.654784441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:40.654343075 +0000 UTC m=+1622.769453192" watchObservedRunningTime="2026-11-04 01:25:40.654784441 +0000 UTC m=+1622.769894558" Nov 04 01:25:40.701774 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:40.701710 4852 patch_prober.go:29] interesting pod/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:25:40.701774 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:40.701757 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" podUID="eef87d33-f22e-4ce2-ad38-94ac8f23913f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 04 01:25:40.746385 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:40.746343 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:41.626781 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:41.626751 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf" event={"ID":"dc338560-6a2d-4fdd-bd5c-23e1071d63a7","Type":"ContainerStarted","Data":"5638aabe5a2a06d784a6e7a60f4e06383f22f77e710868b964b2d41f1a129ebc"} Nov 04 01:25:41.627336 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:41.627309 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf" event={"ID":"dc338560-6a2d-4fdd-bd5c-23e1071d63a7","Type":"ContainerStarted","Data":"ca549dd19edbcbddf3366948189e541d3f8751091b37a84dfa20a08923badacb"} Nov 04 01:25:41.627454 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:41.627342 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf" Nov 04 01:25:41.628274 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:41.628252 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2" event={"ID":"514b08d9-34ac-4867-90a4-53fd518d25c1","Type":"ContainerStarted","Data":"f82e6f14a5c2d9da63031adc4ce3556a77860185175f943f4a8002d44956b5d7"} Nov 04 01:25:41.628363 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:41.628298 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2" event={"ID":"514b08d9-34ac-4867-90a4-53fd518d25c1","Type":"ContainerStarted","Data":"cb0d012b21594d19e37ab88fa30c7184ca88857a5e3180e216c06c605e5265d5"} Nov 04 01:25:41.704589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:41.704495 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:41.706106 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:41.706074 4852 patch_prober.go:29] interesting pod/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:25:41.706318 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:41.706293 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" podUID="eef87d33-f22e-4ce2-ad38-94ac8f23913f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 04 01:25:41.739630 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:41.729425 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf" podStartSLOduration=3.729391547 podStartE2EDuration="3.729391547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:41.67139898 +0000 UTC m=+1623.786509099" watchObservedRunningTime="2026-11-04 01:25:41.729391547 +0000 UTC m=+1623.844501666" Nov 04 01:25:41.755479 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:41.752208 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2" podStartSLOduration=3.752192285 podStartE2EDuration="3.752192285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:41.722338313 +0000 UTC m=+1623.837448451" watchObservedRunningTime="2026-11-04 01:25:41.752192285 +0000 UTC m=+1623.867302406" Nov 04 01:25:42.675593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:42.674934 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:42.709141 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:42.705872 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2" Nov 04 01:25:42.711000 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:42.710670 4852 patch_prober.go:29] interesting pod/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:25:42.711000 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:42.710798 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" podUID="eef87d33-f22e-4ce2-ad38-94ac8f23913f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 04 01:25:43.696132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:43.695890 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:43.708509 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:43.708479 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:43.708885 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:43.708862 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:43.709648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:43.709629 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:44.319725 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.319688 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk"] Nov 04 01:25:44.325128 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.325106 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" Nov 04 01:25:44.335689 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.335108 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-62510d3b-serving-cert\"" Nov 04 01:25:44.338112 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.338082 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk"] Nov 04 01:25:44.348164 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.348131 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70aa5fc4-7c6f-4748-85a0-091d29781f8c-metrics-certs\") pod \"maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk\" (UID: \"70aa5fc4-7c6f-4748-85a0-091d29781f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" Nov 04 01:25:44.348306 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.348193 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70aa5fc4-7c6f-4748-85a0-091d29781f8c-maas-api-tls\") pod \"maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk\" (UID: \"70aa5fc4-7c6f-4748-85a0-091d29781f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" Nov 04 01:25:44.348423 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.348403 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70aa5fc4-7c6f-4748-85a0-091d29781f8c-openshift-service-ca\") pod \"maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk\" (UID: \"70aa5fc4-7c6f-4748-85a0-091d29781f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" Nov 04 01:25:44.348474 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.348436 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmjmq\" (UniqueName: \"kubernetes.io/projected/70aa5fc4-7c6f-4748-85a0-091d29781f8c-kube-api-access-vmjmq\") pod \"maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk\" (UID: \"70aa5fc4-7c6f-4748-85a0-091d29781f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" Nov 04 01:25:44.421947 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.421923 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:44.449908 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.449874 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70aa5fc4-7c6f-4748-85a0-091d29781f8c-openshift-service-ca\") pod \"maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk\" (UID: \"70aa5fc4-7c6f-4748-85a0-091d29781f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" Nov 04 01:25:44.450042 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.449910 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmjmq\" (UniqueName: \"kubernetes.io/projected/70aa5fc4-7c6f-4748-85a0-091d29781f8c-kube-api-access-vmjmq\") pod \"maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk\" (UID: \"70aa5fc4-7c6f-4748-85a0-091d29781f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" Nov 04 01:25:44.450098 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.450068 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70aa5fc4-7c6f-4748-85a0-091d29781f8c-metrics-certs\") pod \"maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk\" (UID: \"70aa5fc4-7c6f-4748-85a0-091d29781f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" Nov 04 01:25:44.450181 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.450108 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70aa5fc4-7c6f-4748-85a0-091d29781f8c-maas-api-tls\") pod \"maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk\" (UID: \"70aa5fc4-7c6f-4748-85a0-091d29781f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" Nov 04 01:25:44.450520 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.450497 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70aa5fc4-7c6f-4748-85a0-091d29781f8c-openshift-service-ca\") pod \"maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk\" (UID: \"70aa5fc4-7c6f-4748-85a0-091d29781f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" Nov 04 01:25:44.452679 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.452661 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70aa5fc4-7c6f-4748-85a0-091d29781f8c-metrics-certs\") pod \"maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk\" (UID: \"70aa5fc4-7c6f-4748-85a0-091d29781f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" Nov 04 01:25:44.452837 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.452819 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70aa5fc4-7c6f-4748-85a0-091d29781f8c-maas-api-tls\") pod \"maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk\" (UID: \"70aa5fc4-7c6f-4748-85a0-091d29781f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" Nov 04 01:25:44.458408 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.458389 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmjmq\" (UniqueName: \"kubernetes.io/projected/70aa5fc4-7c6f-4748-85a0-091d29781f8c-kube-api-access-vmjmq\") pod \"maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk\" (UID: \"70aa5fc4-7c6f-4748-85a0-091d29781f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" Nov 04 01:25:44.572954 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.571098 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:44.642475 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.642445 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" Nov 04 01:25:44.722694 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.722138 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:44.907581 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.907462 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd" podUID="3cc908b7-78ae-44d2-a916-598712dc3a72" containerName="payload-processing" containerID="cri-o://17ce10cf633b7e5688e96c68ca92e3f691a248203ab9af62c1ee907289083147" gracePeriod=25 Nov 04 01:25:44.928744 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:25:44.928709 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70aa5fc4_7c6f_4748_85a0_091d29781f8c.slice/crio-3400e243c96b0d0899f03a63cc5f49c219210b300bd65653574518149f710912 WatchSource:0}: Error finding container 3400e243c96b0d0899f03a63cc5f49c219210b300bd65653574518149f710912: Status 404 returned error can't find the container with id 3400e243c96b0d0899f03a63cc5f49c219210b300bd65653574518149f710912 Nov 04 01:25:44.929637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.929613 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk"] Nov 04 01:25:44.930884 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:44.930851 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5" podUID="bfb9e9da-63a3-4c80-aca8-7600e83ee4dd" containerName="payload-pre-processing" containerID="cri-o://cd5964e2f6e5564402203377277302c01ff6b5531b3ecb0da517905c3cd69772" gracePeriod=25 Nov 04 01:25:45.356523 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.356361 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:45.442654 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.436130 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd" Nov 04 01:25:45.490688 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.489049 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cc908b7-78ae-44d2-a916-598712dc3a72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc908b7-78ae-44d2-a916-598712dc3a72-plugins-config-volume\") pod \"3cc908b7-78ae-44d2-a916-598712dc3a72\" (UID: \"3cc908b7-78ae-44d2-a916-598712dc3a72\") " Nov 04 01:25:45.490688 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.489113 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cc908b7-78ae-44d2-a916-598712dc3a72-kube-api-access-768w6\" (UniqueName: \"kubernetes.io/projected/3cc908b7-78ae-44d2-a916-598712dc3a72-kube-api-access-768w6\") pod \"3cc908b7-78ae-44d2-a916-598712dc3a72\" (UID: \"3cc908b7-78ae-44d2-a916-598712dc3a72\") " Nov 04 01:25:45.493907 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.493867 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cc908b7-78ae-44d2-a916-598712dc3a72-plugins-config-volume" pod "3cc908b7-78ae-44d2-a916-598712dc3a72" (UID: "3cc908b7-78ae-44d2-a916-598712dc3a72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:45.683854 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.675219 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc908b7-78ae-44d2-a916-598712dc3a72-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:45.707665 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.707553 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cc908b7-78ae-44d2-a916-598712dc3a72-kube-api-access-768w6" pod "3cc908b7-78ae-44d2-a916-598712dc3a72" (UID: "3cc908b7-78ae-44d2-a916-598712dc3a72"). InnerVolumeSpecName "kube-api-access-768w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:45.737148 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.737117 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" event={"ID":"70aa5fc4-7c6f-4748-85a0-091d29781f8c","Type":"ContainerStarted","Data":"3400e243c96b0d0899f03a63cc5f49c219210b300bd65653574518149f710912"} Nov 04 01:25:45.737761 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.737737 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5" Nov 04 01:25:45.738861 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.738835 4852 generic.go:356] "Generic (PLEG): container finished" podID="bfb9e9da-63a3-4c80-aca8-7600e83ee4dd" containerID="cd5964e2f6e5564402203377277302c01ff6b5531b3ecb0da517905c3cd69772" exitCode=0 Nov 04 01:25:45.739013 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.738909 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5" event={"ID":"bfb9e9da-63a3-4c80-aca8-7600e83ee4dd","Type":"ContainerDied","Data":"cd5964e2f6e5564402203377277302c01ff6b5531b3ecb0da517905c3cd69772"} Nov 04 01:25:45.739013 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.738969 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5" event={"ID":"bfb9e9da-63a3-4c80-aca8-7600e83ee4dd","Type":"ContainerDied","Data":"28d3c52adabfbf875f9980c9030ef1785f8523ebeb33a4c4bf22cea047202a5c"} Nov 04 01:25:45.739013 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.738994 4852 scope.go:122] "RemoveContainer" containerID="cd5964e2f6e5564402203377277302c01ff6b5531b3ecb0da517905c3cd69772" Nov 04 01:25:45.740506 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.740483 4852 generic.go:356] "Generic (PLEG): container finished" podID="3cc908b7-78ae-44d2-a916-598712dc3a72" containerID="17ce10cf633b7e5688e96c68ca92e3f691a248203ab9af62c1ee907289083147" exitCode=0 Nov 04 01:25:45.740651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.740518 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd" event={"ID":"3cc908b7-78ae-44d2-a916-598712dc3a72","Type":"ContainerDied","Data":"17ce10cf633b7e5688e96c68ca92e3f691a248203ab9af62c1ee907289083147"} Nov 04 01:25:45.740715 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.740529 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd" Nov 04 01:25:45.740715 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.740665 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd" event={"ID":"3cc908b7-78ae-44d2-a916-598712dc3a72","Type":"ContainerDied","Data":"b510f6e01f69995a4fe0d79b47b78667b88f01315d1456a9b35524915dca1176"} Nov 04 01:25:45.756404 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.756340 4852 scope.go:122] "RemoveContainer" containerID="cd5964e2f6e5564402203377277302c01ff6b5531b3ecb0da517905c3cd69772" Nov 04 01:25:45.756663 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:25:45.756643 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd5964e2f6e5564402203377277302c01ff6b5531b3ecb0da517905c3cd69772\": container with ID starting with cd5964e2f6e5564402203377277302c01ff6b5531b3ecb0da517905c3cd69772 not found: ID does not exist" containerID="cd5964e2f6e5564402203377277302c01ff6b5531b3ecb0da517905c3cd69772" Nov 04 01:25:45.756730 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.756672 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd5964e2f6e5564402203377277302c01ff6b5531b3ecb0da517905c3cd69772"} err="failed to get container status \"cd5964e2f6e5564402203377277302c01ff6b5531b3ecb0da517905c3cd69772\": rpc error: code = NotFound desc = could not find container \"cd5964e2f6e5564402203377277302c01ff6b5531b3ecb0da517905c3cd69772\": container with ID starting with cd5964e2f6e5564402203377277302c01ff6b5531b3ecb0da517905c3cd69772 not found: ID does not exist" Nov 04 01:25:45.756730 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.756689 4852 scope.go:122] "RemoveContainer" containerID="17ce10cf633b7e5688e96c68ca92e3f691a248203ab9af62c1ee907289083147" Nov 04 01:25:45.768642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.768623 4852 scope.go:122] "RemoveContainer" containerID="17ce10cf633b7e5688e96c68ca92e3f691a248203ab9af62c1ee907289083147" Nov 04 01:25:45.768936 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:25:45.768915 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ce10cf633b7e5688e96c68ca92e3f691a248203ab9af62c1ee907289083147\": container with ID starting with 17ce10cf633b7e5688e96c68ca92e3f691a248203ab9af62c1ee907289083147 not found: ID does not exist" containerID="17ce10cf633b7e5688e96c68ca92e3f691a248203ab9af62c1ee907289083147" Nov 04 01:25:45.769006 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.768948 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ce10cf633b7e5688e96c68ca92e3f691a248203ab9af62c1ee907289083147"} err="failed to get container status \"17ce10cf633b7e5688e96c68ca92e3f691a248203ab9af62c1ee907289083147\": rpc error: code = NotFound desc = could not find container \"17ce10cf633b7e5688e96c68ca92e3f691a248203ab9af62c1ee907289083147\": container with ID starting with 17ce10cf633b7e5688e96c68ca92e3f691a248203ab9af62c1ee907289083147 not found: ID does not exist" Nov 04 01:25:45.777558 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.777532 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-768w6\" (UniqueName: \"kubernetes.io/projected/3cc908b7-78ae-44d2-a916-598712dc3a72-kube-api-access-768w6\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:45.827671 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.827639 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd"] Nov 04 01:25:45.831972 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.831944 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7024f0c-546f99ff6b-zjwsd"] Nov 04 01:25:45.878600 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.878537 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfb9e9da-63a3-4c80-aca8-7600e83ee4dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb9e9da-63a3-4c80-aca8-7600e83ee4dd-plugins-config-volume\") pod \"bfb9e9da-63a3-4c80-aca8-7600e83ee4dd\" (UID: \"bfb9e9da-63a3-4c80-aca8-7600e83ee4dd\") " Nov 04 01:25:45.878822 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.878772 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfb9e9da-63a3-4c80-aca8-7600e83ee4dd-kube-api-access-ks29l\" (UniqueName: \"kubernetes.io/projected/bfb9e9da-63a3-4c80-aca8-7600e83ee4dd-kube-api-access-ks29l\") pod \"bfb9e9da-63a3-4c80-aca8-7600e83ee4dd\" (UID: \"bfb9e9da-63a3-4c80-aca8-7600e83ee4dd\") " Nov 04 01:25:45.879354 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.878985 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb9e9da-63a3-4c80-aca8-7600e83ee4dd-plugins-config-volume" pod "bfb9e9da-63a3-4c80-aca8-7600e83ee4dd" (UID: "bfb9e9da-63a3-4c80-aca8-7600e83ee4dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:45.880245 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.880219 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb9e9da-63a3-4c80-aca8-7600e83ee4dd-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:45.882451 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.882422 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb9e9da-63a3-4c80-aca8-7600e83ee4dd-kube-api-access-ks29l" pod "bfb9e9da-63a3-4c80-aca8-7600e83ee4dd" (UID: "bfb9e9da-63a3-4c80-aca8-7600e83ee4dd"). InnerVolumeSpecName "kube-api-access-ks29l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:45.981477 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.981390 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks29l\" (UniqueName: \"kubernetes.io/projected/bfb9e9da-63a3-4c80-aca8-7600e83ee4dd-kube-api-access-ks29l\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:45.995660 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:45.995628 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cc908b7-78ae-44d2-a916-598712dc3a72" path="/var/lib/kubelet/pods/3cc908b7-78ae-44d2-a916-598712dc3a72/volumes" Nov 04 01:25:46.107610 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.107574 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb"] Nov 04 01:25:46.108184 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.108169 4852 cpu_manager.go:414] "containerMap: removing container" podUID="3cc908b7-78ae-44d2-a916-598712dc3a72" containerName="payload-processing" Nov 04 01:25:46.108236 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.108184 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cc908b7-78ae-44d2-a916-598712dc3a72" containerName="payload-processing" Nov 04 01:25:46.108236 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.108197 4852 cpu_manager.go:414] "containerMap: removing container" podUID="bfb9e9da-63a3-4c80-aca8-7600e83ee4dd" containerName="payload-pre-processing" Nov 04 01:25:46.108236 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.108203 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfb9e9da-63a3-4c80-aca8-7600e83ee4dd" containerName="payload-pre-processing" Nov 04 01:25:46.108330 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.108302 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cc908b7-78ae-44d2-a916-598712dc3a72" containerName="payload-processing" Nov 04 01:25:46.108330 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.108309 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfb9e9da-63a3-4c80-aca8-7600e83ee4dd" containerName="payload-pre-processing" Nov 04 01:25:46.112018 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.112001 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" Nov 04 01:25:46.118039 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.118011 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-296c124f-serving-cert\"" Nov 04 01:25:46.126008 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.125982 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb"] Nov 04 01:25:46.183973 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.183941 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87443b1-c333-4d70-9843-e604b060f596-openshift-service-ca\") pod \"maas-api-e2e-ait-296c124f-f68689bf6-qg9mb\" (UID: \"e87443b1-c333-4d70-9843-e604b060f596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" Nov 04 01:25:46.183973 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.183977 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e87443b1-c333-4d70-9843-e604b060f596-maas-api-tls\") pod \"maas-api-e2e-ait-296c124f-f68689bf6-qg9mb\" (UID: \"e87443b1-c333-4d70-9843-e604b060f596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" Nov 04 01:25:46.184197 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.184149 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glx77\" (UniqueName: \"kubernetes.io/projected/e87443b1-c333-4d70-9843-e604b060f596-kube-api-access-glx77\") pod \"maas-api-e2e-ait-296c124f-f68689bf6-qg9mb\" (UID: \"e87443b1-c333-4d70-9843-e604b060f596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" Nov 04 01:25:46.184296 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.184283 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e87443b1-c333-4d70-9843-e604b060f596-metrics-certs\") pod \"maas-api-e2e-ait-296c124f-f68689bf6-qg9mb\" (UID: \"e87443b1-c333-4d70-9843-e604b060f596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" Nov 04 01:25:46.212524 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.212494 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:46.287740 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.287705 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glx77\" (UniqueName: \"kubernetes.io/projected/e87443b1-c333-4d70-9843-e604b060f596-kube-api-access-glx77\") pod \"maas-api-e2e-ait-296c124f-f68689bf6-qg9mb\" (UID: \"e87443b1-c333-4d70-9843-e604b060f596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" Nov 04 01:25:46.289006 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.288981 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e87443b1-c333-4d70-9843-e604b060f596-metrics-certs\") pod \"maas-api-e2e-ait-296c124f-f68689bf6-qg9mb\" (UID: \"e87443b1-c333-4d70-9843-e604b060f596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" Nov 04 01:25:46.298583 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.289294 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87443b1-c333-4d70-9843-e604b060f596-openshift-service-ca\") pod \"maas-api-e2e-ait-296c124f-f68689bf6-qg9mb\" (UID: \"e87443b1-c333-4d70-9843-e604b060f596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" Nov 04 01:25:46.298890 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.298862 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e87443b1-c333-4d70-9843-e604b060f596-maas-api-tls\") pod \"maas-api-e2e-ait-296c124f-f68689bf6-qg9mb\" (UID: \"e87443b1-c333-4d70-9843-e604b060f596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" Nov 04 01:25:46.311245 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.302192 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e87443b1-c333-4d70-9843-e604b060f596-metrics-certs\") pod \"maas-api-e2e-ait-296c124f-f68689bf6-qg9mb\" (UID: \"e87443b1-c333-4d70-9843-e604b060f596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" Nov 04 01:25:46.311245 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.291770 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87443b1-c333-4d70-9843-e604b060f596-openshift-service-ca\") pod \"maas-api-e2e-ait-296c124f-f68689bf6-qg9mb\" (UID: \"e87443b1-c333-4d70-9843-e604b060f596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" Nov 04 01:25:46.311245 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.308134 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e87443b1-c333-4d70-9843-e604b060f596-maas-api-tls\") pod \"maas-api-e2e-ait-296c124f-f68689bf6-qg9mb\" (UID: \"e87443b1-c333-4d70-9843-e604b060f596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" Nov 04 01:25:46.311245 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.309770 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glx77\" (UniqueName: \"kubernetes.io/projected/e87443b1-c333-4d70-9843-e604b060f596-kube-api-access-glx77\") pod \"maas-api-e2e-ait-296c124f-f68689bf6-qg9mb\" (UID: \"e87443b1-c333-4d70-9843-e604b060f596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" Nov 04 01:25:46.363477 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.363442 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:46.429100 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.429070 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" Nov 04 01:25:46.522777 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.522744 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:46.817860 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.817828 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh"] Nov 04 01:25:46.818666 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.818635 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" podUID="5972b292-b0b7-45af-9674-f0c00f20c1c9" containerName="maas-api" containerID="cri-o://3b28ca4b4f5964c01b126dc0576dd3e135325b25265a9f495c8875425207e790" gracePeriod=30 Nov 04 01:25:46.903094 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.887853 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" event={"ID":"70aa5fc4-7c6f-4748-85a0-091d29781f8c","Type":"ContainerStarted","Data":"3fd0608c52d2e8dcba4ba26642846ba56cae585755971659d4a5c08bb6a68181"} Nov 04 01:25:46.903094 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.889086 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" Nov 04 01:25:46.916363 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.913982 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5" Nov 04 01:25:46.994641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:46.986843 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb"] Nov 04 01:25:47.042088 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.042051 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:47.060734 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.060053 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" podStartSLOduration=2.334544399 podStartE2EDuration="3.060032554s" totalImagesPullingTime="725.488155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:46.969116217 +0000 UTC m=+1629.084226336" watchObservedRunningTime="2026-11-04 01:25:47.060032554 +0000 UTC m=+1629.175142672" Nov 04 01:25:47.086916 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.086815 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5"] Nov 04 01:25:47.094412 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.094377 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7024f0c-6bbf47f699-z6lr5"] Nov 04 01:25:47.270451 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.270416 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5"] Nov 04 01:25:47.270897 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.270868 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" podUID="eef87d33-f22e-4ce2-ad38-94ac8f23913f" containerName="istio-proxy" containerID="cri-o://49e5c5d885267acc74698223540a0a17f037c410cee3ef230dc552061e0bdc96" gracePeriod=30 Nov 04 01:25:47.287393 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.287362 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:47.544102 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.544068 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:47.744469 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.744445 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" Nov 04 01:25:47.829215 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.829191 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5972b292-b0b7-45af-9674-f0c00f20c1c9-kube-api-access-s84rr\" (UniqueName: \"kubernetes.io/projected/5972b292-b0b7-45af-9674-f0c00f20c1c9-kube-api-access-s84rr\") pod \"5972b292-b0b7-45af-9674-f0c00f20c1c9\" (UID: \"5972b292-b0b7-45af-9674-f0c00f20c1c9\") " Nov 04 01:25:47.829639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.829235 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5972b292-b0b7-45af-9674-f0c00f20c1c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5972b292-b0b7-45af-9674-f0c00f20c1c9-openshift-service-ca\") pod \"5972b292-b0b7-45af-9674-f0c00f20c1c9\" (UID: \"5972b292-b0b7-45af-9674-f0c00f20c1c9\") " Nov 04 01:25:47.829639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.829305 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-metrics-certs\") pod \"5972b292-b0b7-45af-9674-f0c00f20c1c9\" (UID: \"5972b292-b0b7-45af-9674-f0c00f20c1c9\") " Nov 04 01:25:47.829639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.829457 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-maas-api-tls\") pod \"5972b292-b0b7-45af-9674-f0c00f20c1c9\" (UID: \"5972b292-b0b7-45af-9674-f0c00f20c1c9\") " Nov 04 01:25:47.829896 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.829682 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5972b292-b0b7-45af-9674-f0c00f20c1c9-openshift-service-ca" pod "5972b292-b0b7-45af-9674-f0c00f20c1c9" (UID: "5972b292-b0b7-45af-9674-f0c00f20c1c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:47.830404 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.830373 4852 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5972b292-b0b7-45af-9674-f0c00f20c1c9-openshift-service-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:47.831749 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.831722 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-metrics-certs" pod "5972b292-b0b7-45af-9674-f0c00f20c1c9" (UID: "5972b292-b0b7-45af-9674-f0c00f20c1c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:25:47.831749 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.831740 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-maas-api-tls" pod "5972b292-b0b7-45af-9674-f0c00f20c1c9" (UID: "5972b292-b0b7-45af-9674-f0c00f20c1c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:25:47.831867 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.831829 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5972b292-b0b7-45af-9674-f0c00f20c1c9-kube-api-access-s84rr" pod "5972b292-b0b7-45af-9674-f0c00f20c1c9" (UID: "5972b292-b0b7-45af-9674-f0c00f20c1c9"). InnerVolumeSpecName "kube-api-access-s84rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:47.921858 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.921818 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" event={"ID":"e87443b1-c333-4d70-9843-e604b060f596","Type":"ContainerStarted","Data":"c34f41912208cbe8d8f260a26b17dc41f6595585a46501d9996756692052c104"} Nov 04 01:25:47.922045 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.921914 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" event={"ID":"e87443b1-c333-4d70-9843-e604b060f596","Type":"ContainerStarted","Data":"8db053f5c7dcafd8d4190bc54521420daec2583d46be20ae79a74801114e4f70"} Nov 04 01:25:47.922045 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.921942 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" Nov 04 01:25:47.923237 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.923213 4852 generic.go:356] "Generic (PLEG): container finished" podID="5972b292-b0b7-45af-9674-f0c00f20c1c9" containerID="3b28ca4b4f5964c01b126dc0576dd3e135325b25265a9f495c8875425207e790" exitCode=0 Nov 04 01:25:47.923344 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.923249 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" Nov 04 01:25:47.923344 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.923293 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" event={"ID":"5972b292-b0b7-45af-9674-f0c00f20c1c9","Type":"ContainerDied","Data":"3b28ca4b4f5964c01b126dc0576dd3e135325b25265a9f495c8875425207e790"} Nov 04 01:25:47.923344 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.923337 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh" event={"ID":"5972b292-b0b7-45af-9674-f0c00f20c1c9","Type":"ContainerDied","Data":"7adf7924e69b17be9f380aae0b20ef763ec9b71f7ef3ebaf3671340db231760f"} Nov 04 01:25:47.923441 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.923358 4852 scope.go:122] "RemoveContainer" containerID="3b28ca4b4f5964c01b126dc0576dd3e135325b25265a9f495c8875425207e790" Nov 04 01:25:47.931222 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.931199 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-maas-api-tls\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:47.931222 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.931223 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s84rr\" (UniqueName: \"kubernetes.io/projected/5972b292-b0b7-45af-9674-f0c00f20c1c9-kube-api-access-s84rr\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:47.931222 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.931236 4852 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5972b292-b0b7-45af-9674-f0c00f20c1c9-metrics-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:47.934434 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.934416 4852 scope.go:122] "RemoveContainer" containerID="3b28ca4b4f5964c01b126dc0576dd3e135325b25265a9f495c8875425207e790" Nov 04 01:25:47.934724 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:25:47.934704 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b28ca4b4f5964c01b126dc0576dd3e135325b25265a9f495c8875425207e790\": container with ID starting with 3b28ca4b4f5964c01b126dc0576dd3e135325b25265a9f495c8875425207e790 not found: ID does not exist" containerID="3b28ca4b4f5964c01b126dc0576dd3e135325b25265a9f495c8875425207e790" Nov 04 01:25:47.934772 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.934732 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b28ca4b4f5964c01b126dc0576dd3e135325b25265a9f495c8875425207e790"} err="failed to get container status \"3b28ca4b4f5964c01b126dc0576dd3e135325b25265a9f495c8875425207e790\": rpc error: code = NotFound desc = could not find container \"3b28ca4b4f5964c01b126dc0576dd3e135325b25265a9f495c8875425207e790\": container with ID starting with 3b28ca4b4f5964c01b126dc0576dd3e135325b25265a9f495c8875425207e790 not found: ID does not exist" Nov 04 01:25:47.946844 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:47.946802 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" podStartSLOduration=1.317699044 podStartE2EDuration="1.946791701s" totalImagesPullingTime="629.092657ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:47.944373674 +0000 UTC m=+1630.059483791" watchObservedRunningTime="2026-11-04 01:25:47.946791701 +0000 UTC m=+1630.061901819" Nov 04 01:25:48.002879 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:48.002848 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfb9e9da-63a3-4c80-aca8-7600e83ee4dd" path="/var/lib/kubelet/pods/bfb9e9da-63a3-4c80-aca8-7600e83ee4dd/volumes" Nov 04 01:25:48.003204 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:48.003187 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh"] Nov 04 01:25:48.003299 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:48.003225 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7024f0c-9866cd6cf-mmdxh"] Nov 04 01:25:48.329211 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:48.329170 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:48.532715 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:48.532665 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:49.267099 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:49.267060 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:49.995724 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:49.995690 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5972b292-b0b7-45af-9674-f0c00f20c1c9" path="/var/lib/kubelet/pods/5972b292-b0b7-45af-9674-f0c00f20c1c9/volumes" Nov 04 01:25:50.085593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:50.084707 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:50.827396 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:50.827319 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:51.535596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:51.534545 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:51.631579 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:51.631544 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb" Nov 04 01:25:52.287587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.287099 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:52.561246 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.561217 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:52.708635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.708604 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf" Nov 04 01:25:52.709524 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.709502 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-workload-certs\") pod \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " Nov 04 01:25:52.709671 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.709540 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-envoy\") pod \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " Nov 04 01:25:52.709671 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.709663 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istiod-ca-cert\") pod \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " Nov 04 01:25:52.709792 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.709682 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-token\" (UniqueName: \"kubernetes.io/projected/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-token\") pod \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " Nov 04 01:25:52.709792 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.709711 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-podinfo\") pod \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " Nov 04 01:25:52.709792 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.709730 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eef87d33-f22e-4ce2-ad38-94ac8f23913f-kube-api-access-mqlf5\" (UniqueName: \"kubernetes.io/projected/eef87d33-f22e-4ce2-ad38-94ac8f23913f-kube-api-access-mqlf5\") pod \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " Nov 04 01:25:52.709792 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.709756 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-credential-socket\") pod \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " Nov 04 01:25:52.710003 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.709824 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-workload-socket\") pod \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " Nov 04 01:25:52.710003 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.709936 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-data\") pod \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\" (UID: \"eef87d33-f22e-4ce2-ad38-94ac8f23913f\") " Nov 04 01:25:52.710466 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.710445 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-data" pod "eef87d33-f22e-4ce2-ad38-94ac8f23913f" (UID: "eef87d33-f22e-4ce2-ad38-94ac8f23913f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:52.710989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.710957 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-credential-socket" pod "eef87d33-f22e-4ce2-ad38-94ac8f23913f" (UID: "eef87d33-f22e-4ce2-ad38-94ac8f23913f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:52.711089 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.711066 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istiod-ca-cert" pod "eef87d33-f22e-4ce2-ad38-94ac8f23913f" (UID: "eef87d33-f22e-4ce2-ad38-94ac8f23913f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:52.711174 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.711155 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-workload-certs" pod "eef87d33-f22e-4ce2-ad38-94ac8f23913f" (UID: "eef87d33-f22e-4ce2-ad38-94ac8f23913f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:52.711270 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.711175 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-workload-socket" pod "eef87d33-f22e-4ce2-ad38-94ac8f23913f" (UID: "eef87d33-f22e-4ce2-ad38-94ac8f23913f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:52.711717 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.711697 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q" Nov 04 01:25:52.713630 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.713610 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-token" pod "eef87d33-f22e-4ce2-ad38-94ac8f23913f" (UID: "eef87d33-f22e-4ce2-ad38-94ac8f23913f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:52.714711 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.714691 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef87d33-f22e-4ce2-ad38-94ac8f23913f-kube-api-access-mqlf5" pod "eef87d33-f22e-4ce2-ad38-94ac8f23913f" (UID: "eef87d33-f22e-4ce2-ad38-94ac8f23913f"). InnerVolumeSpecName "kube-api-access-mqlf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:52.714938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.714913 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-podinfo" pod "eef87d33-f22e-4ce2-ad38-94ac8f23913f" (UID: "eef87d33-f22e-4ce2-ad38-94ac8f23913f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:25:52.715617 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.715603 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-envoy" pod "eef87d33-f22e-4ce2-ad38-94ac8f23913f" (UID: "eef87d33-f22e-4ce2-ad38-94ac8f23913f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:52.812024 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.811939 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:52.812024 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.811985 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:52.812024 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.812004 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:52.812024 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.812015 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqlf5\" (UniqueName: \"kubernetes.io/projected/eef87d33-f22e-4ce2-ad38-94ac8f23913f-kube-api-access-mqlf5\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:52.812024 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.812024 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:52.812284 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.812033 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:52.812284 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.812042 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:52.812284 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.812050 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:52.812284 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.812059 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef87d33-f22e-4ce2-ad38-94ac8f23913f-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:52.958452 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.958415 4852 generic.go:356] "Generic (PLEG): container finished" podID="eef87d33-f22e-4ce2-ad38-94ac8f23913f" containerID="49e5c5d885267acc74698223540a0a17f037c410cee3ef230dc552061e0bdc96" exitCode=0 Nov 04 01:25:52.958687 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.958461 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" Nov 04 01:25:52.958687 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.958490 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" event={"ID":"eef87d33-f22e-4ce2-ad38-94ac8f23913f","Type":"ContainerDied","Data":"49e5c5d885267acc74698223540a0a17f037c410cee3ef230dc552061e0bdc96"} Nov 04 01:25:52.958687 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.958556 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5" event={"ID":"eef87d33-f22e-4ce2-ad38-94ac8f23913f","Type":"ContainerDied","Data":"e0c140e33677d6de643641e90ef6064a806fc27e812e24f17c98240aa911a9a7"} Nov 04 01:25:52.958687 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.958612 4852 scope.go:122] "RemoveContainer" containerID="49e5c5d885267acc74698223540a0a17f037c410cee3ef230dc552061e0bdc96" Nov 04 01:25:52.970624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.970607 4852 scope.go:122] "RemoveContainer" containerID="49e5c5d885267acc74698223540a0a17f037c410cee3ef230dc552061e0bdc96" Nov 04 01:25:52.970865 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:25:52.970850 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49e5c5d885267acc74698223540a0a17f037c410cee3ef230dc552061e0bdc96\": container with ID starting with 49e5c5d885267acc74698223540a0a17f037c410cee3ef230dc552061e0bdc96 not found: ID does not exist" containerID="49e5c5d885267acc74698223540a0a17f037c410cee3ef230dc552061e0bdc96" Nov 04 01:25:52.970910 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.970873 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49e5c5d885267acc74698223540a0a17f037c410cee3ef230dc552061e0bdc96"} err="failed to get container status \"49e5c5d885267acc74698223540a0a17f037c410cee3ef230dc552061e0bdc96\": rpc error: code = NotFound desc = could not find container \"49e5c5d885267acc74698223540a0a17f037c410cee3ef230dc552061e0bdc96\": container with ID starting with 49e5c5d885267acc74698223540a0a17f037c410cee3ef230dc552061e0bdc96 not found: ID does not exist" Nov 04 01:25:52.991248 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.991226 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5"] Nov 04 01:25:52.999538 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:52.999517 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-7vvj5"] Nov 04 01:25:53.256430 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:53.250267 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:53.715495 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:53.715416 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2" Nov 04 01:25:53.966677 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:53.963746 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" Nov 04 01:25:53.971647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:53.968799 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" Nov 04 01:25:54.002370 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:53.999254 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:54.027827 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:54.025695 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eef87d33-f22e-4ce2-ad38-94ac8f23913f" path="/var/lib/kubelet/pods/eef87d33-f22e-4ce2-ad38-94ac8f23913f/volumes" Nov 04 01:25:54.701094 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:54.700782 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:55.676588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:55.675024 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:56.662588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:56.648788 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:57.362878 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:57.361830 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:58.157593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:58.156050 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:58.882085 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:58.882052 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:59.549518 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:25:59.549483 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:59.866093 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:25:59.866006 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:26:00.223433 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:00.223338 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:00.977731 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:00.977696 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:01.870033 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:01.870000 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:02.772114 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:02.758295 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:03.528140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:03.528103 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:03.690729 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:03.690700 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:26:04.495684 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:04.495649 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:04.700617 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:04.700584 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:05.437271 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:05.437235 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:06.595272 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:06.595231 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:26:06.595793 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:06.595744 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" podUID="baefa3c0-28be-428a-9591-f0ca1f07fafa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:07.426076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:07.426026 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:08.051181 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:08.051153 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2"] Nov 04 01:26:08.110593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:08.110540 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf"] Nov 04 01:26:08.243673 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:08.243601 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:08.441306 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:08.441220 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:08.630144 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:08.630086 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:09.783242 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:09.783189 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:10.562849 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:10.562811 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:11.330166 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:11.330134 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:11.589812 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:11.589688 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:12.174596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:12.170044 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:12.590969 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:12.590939 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb"] Nov 04 01:26:12.591832 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:12.591557 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" podUID="e87443b1-c333-4d70-9843-e604b060f596" containerName="maas-api" containerID="cri-o://c34f41912208cbe8d8f260a26b17dc41f6595585a46501d9996756692052c104" gracePeriod=30 Nov 04 01:26:12.684706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:12.684678 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:12.853909 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:12.853689 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:13.088690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.087690 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2" podUID="514b08d9-34ac-4867-90a4-53fd518d25c1" containerName="payload-processing" containerID="cri-o://f82e6f14a5c2d9da63031adc4ce3556a77860185175f943f4a8002d44956b5d7" gracePeriod=25 Nov 04 01:26:13.089604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.089461 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:13.111473 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.111190 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf" podUID="dc338560-6a2d-4fdd-bd5c-23e1071d63a7" containerName="payload-pre-processing" containerID="cri-o://5638aabe5a2a06d784a6e7a60f4e06383f22f77e710868b964b2d41f1a129ebc" gracePeriod=25 Nov 04 01:26:13.334859 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.334496 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" Nov 04 01:26:13.435706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.435622 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e87443b1-c333-4d70-9843-e604b060f596-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87443b1-c333-4d70-9843-e604b060f596-openshift-service-ca\") pod \"e87443b1-c333-4d70-9843-e604b060f596\" (UID: \"e87443b1-c333-4d70-9843-e604b060f596\") " Nov 04 01:26:13.435869 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.435709 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e87443b1-c333-4d70-9843-e604b060f596-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e87443b1-c333-4d70-9843-e604b060f596-metrics-certs\") pod \"e87443b1-c333-4d70-9843-e604b060f596\" (UID: \"e87443b1-c333-4d70-9843-e604b060f596\") " Nov 04 01:26:13.435869 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.435782 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e87443b1-c333-4d70-9843-e604b060f596-kube-api-access-glx77\" (UniqueName: \"kubernetes.io/projected/e87443b1-c333-4d70-9843-e604b060f596-kube-api-access-glx77\") pod \"e87443b1-c333-4d70-9843-e604b060f596\" (UID: \"e87443b1-c333-4d70-9843-e604b060f596\") " Nov 04 01:26:13.435869 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.435826 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e87443b1-c333-4d70-9843-e604b060f596-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e87443b1-c333-4d70-9843-e604b060f596-maas-api-tls\") pod \"e87443b1-c333-4d70-9843-e604b060f596\" (UID: \"e87443b1-c333-4d70-9843-e604b060f596\") " Nov 04 01:26:13.436056 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.436028 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e87443b1-c333-4d70-9843-e604b060f596-openshift-service-ca" pod "e87443b1-c333-4d70-9843-e604b060f596" (UID: "e87443b1-c333-4d70-9843-e604b060f596"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:13.436515 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.436446 4852 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87443b1-c333-4d70-9843-e604b060f596-openshift-service-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:13.442337 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.441919 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87443b1-c333-4d70-9843-e604b060f596-kube-api-access-glx77" pod "e87443b1-c333-4d70-9843-e604b060f596" (UID: "e87443b1-c333-4d70-9843-e604b060f596"). InnerVolumeSpecName "kube-api-access-glx77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:13.446944 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.446918 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e87443b1-c333-4d70-9843-e604b060f596-maas-api-tls" pod "e87443b1-c333-4d70-9843-e604b060f596" (UID: "e87443b1-c333-4d70-9843-e604b060f596"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:26:13.447180 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.447150 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e87443b1-c333-4d70-9843-e604b060f596-metrics-certs" pod "e87443b1-c333-4d70-9843-e604b060f596" (UID: "e87443b1-c333-4d70-9843-e604b060f596"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:26:13.452502 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.452477 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2" Nov 04 01:26:13.462311 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.462293 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf" Nov 04 01:26:13.489722 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.472913 4852 generic.go:356] "Generic (PLEG): container finished" podID="dc338560-6a2d-4fdd-bd5c-23e1071d63a7" containerID="5638aabe5a2a06d784a6e7a60f4e06383f22f77e710868b964b2d41f1a129ebc" exitCode=0 Nov 04 01:26:13.489722 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.473042 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf" event={"ID":"dc338560-6a2d-4fdd-bd5c-23e1071d63a7","Type":"ContainerDied","Data":"5638aabe5a2a06d784a6e7a60f4e06383f22f77e710868b964b2d41f1a129ebc"} Nov 04 01:26:13.489722 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.473090 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf" event={"ID":"dc338560-6a2d-4fdd-bd5c-23e1071d63a7","Type":"ContainerDied","Data":"ca549dd19edbcbddf3366948189e541d3f8751091b37a84dfa20a08923badacb"} Nov 04 01:26:13.489722 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.473120 4852 scope.go:122] "RemoveContainer" containerID="5638aabe5a2a06d784a6e7a60f4e06383f22f77e710868b964b2d41f1a129ebc" Nov 04 01:26:13.489722 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.473282 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf" Nov 04 01:26:13.500438 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.500405 4852 generic.go:356] "Generic (PLEG): container finished" podID="514b08d9-34ac-4867-90a4-53fd518d25c1" containerID="f82e6f14a5c2d9da63031adc4ce3556a77860185175f943f4a8002d44956b5d7" exitCode=0 Nov 04 01:26:13.500624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.500602 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2" event={"ID":"514b08d9-34ac-4867-90a4-53fd518d25c1","Type":"ContainerDied","Data":"f82e6f14a5c2d9da63031adc4ce3556a77860185175f943f4a8002d44956b5d7"} Nov 04 01:26:13.500707 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.500662 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2" event={"ID":"514b08d9-34ac-4867-90a4-53fd518d25c1","Type":"ContainerDied","Data":"cb0d012b21594d19e37ab88fa30c7184ca88857a5e3180e216c06c605e5265d5"} Nov 04 01:26:13.500765 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.500754 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2" Nov 04 01:26:13.506137 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.506109 4852 generic.go:356] "Generic (PLEG): container finished" podID="e87443b1-c333-4d70-9843-e604b060f596" containerID="c34f41912208cbe8d8f260a26b17dc41f6595585a46501d9996756692052c104" exitCode=0 Nov 04 01:26:13.506269 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.506231 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" event={"ID":"e87443b1-c333-4d70-9843-e604b060f596","Type":"ContainerDied","Data":"c34f41912208cbe8d8f260a26b17dc41f6595585a46501d9996756692052c104"} Nov 04 01:26:13.506332 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.506281 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" event={"ID":"e87443b1-c333-4d70-9843-e604b060f596","Type":"ContainerDied","Data":"8db053f5c7dcafd8d4190bc54521420daec2583d46be20ae79a74801114e4f70"} Nov 04 01:26:13.506392 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.506360 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb" Nov 04 01:26:13.516604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.516528 4852 scope.go:122] "RemoveContainer" containerID="5638aabe5a2a06d784a6e7a60f4e06383f22f77e710868b964b2d41f1a129ebc" Nov 04 01:26:13.517138 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:26:13.517115 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5638aabe5a2a06d784a6e7a60f4e06383f22f77e710868b964b2d41f1a129ebc\": container with ID starting with 5638aabe5a2a06d784a6e7a60f4e06383f22f77e710868b964b2d41f1a129ebc not found: ID does not exist" containerID="5638aabe5a2a06d784a6e7a60f4e06383f22f77e710868b964b2d41f1a129ebc" Nov 04 01:26:13.517237 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.517145 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5638aabe5a2a06d784a6e7a60f4e06383f22f77e710868b964b2d41f1a129ebc"} err="failed to get container status \"5638aabe5a2a06d784a6e7a60f4e06383f22f77e710868b964b2d41f1a129ebc\": rpc error: code = NotFound desc = could not find container \"5638aabe5a2a06d784a6e7a60f4e06383f22f77e710868b964b2d41f1a129ebc\": container with ID starting with 5638aabe5a2a06d784a6e7a60f4e06383f22f77e710868b964b2d41f1a129ebc not found: ID does not exist" Nov 04 01:26:13.517237 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.517163 4852 scope.go:122] "RemoveContainer" containerID="f82e6f14a5c2d9da63031adc4ce3556a77860185175f943f4a8002d44956b5d7" Nov 04 01:26:13.528776 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.528759 4852 scope.go:122] "RemoveContainer" containerID="f82e6f14a5c2d9da63031adc4ce3556a77860185175f943f4a8002d44956b5d7" Nov 04 01:26:13.529284 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:26:13.529256 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f82e6f14a5c2d9da63031adc4ce3556a77860185175f943f4a8002d44956b5d7\": container with ID starting with f82e6f14a5c2d9da63031adc4ce3556a77860185175f943f4a8002d44956b5d7 not found: ID does not exist" containerID="f82e6f14a5c2d9da63031adc4ce3556a77860185175f943f4a8002d44956b5d7" Nov 04 01:26:13.529442 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.529292 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f82e6f14a5c2d9da63031adc4ce3556a77860185175f943f4a8002d44956b5d7"} err="failed to get container status \"f82e6f14a5c2d9da63031adc4ce3556a77860185175f943f4a8002d44956b5d7\": rpc error: code = NotFound desc = could not find container \"f82e6f14a5c2d9da63031adc4ce3556a77860185175f943f4a8002d44956b5d7\": container with ID starting with f82e6f14a5c2d9da63031adc4ce3556a77860185175f943f4a8002d44956b5d7 not found: ID does not exist" Nov 04 01:26:13.529442 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.529313 4852 scope.go:122] "RemoveContainer" containerID="c34f41912208cbe8d8f260a26b17dc41f6595585a46501d9996756692052c104" Nov 04 01:26:13.537432 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.537399 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc338560-6a2d-4fdd-bd5c-23e1071d63a7-kube-api-access-vv2rl\" (UniqueName: \"kubernetes.io/projected/dc338560-6a2d-4fdd-bd5c-23e1071d63a7-kube-api-access-vv2rl\") pod \"dc338560-6a2d-4fdd-bd5c-23e1071d63a7\" (UID: \"dc338560-6a2d-4fdd-bd5c-23e1071d63a7\") " Nov 04 01:26:13.537595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.537466 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc338560-6a2d-4fdd-bd5c-23e1071d63a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc338560-6a2d-4fdd-bd5c-23e1071d63a7-plugins-config-volume\") pod \"dc338560-6a2d-4fdd-bd5c-23e1071d63a7\" (UID: \"dc338560-6a2d-4fdd-bd5c-23e1071d63a7\") " Nov 04 01:26:13.537595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.537504 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/514b08d9-34ac-4867-90a4-53fd518d25c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514b08d9-34ac-4867-90a4-53fd518d25c1-plugins-config-volume\") pod \"514b08d9-34ac-4867-90a4-53fd518d25c1\" (UID: \"514b08d9-34ac-4867-90a4-53fd518d25c1\") " Nov 04 01:26:13.537595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.537537 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/514b08d9-34ac-4867-90a4-53fd518d25c1-kube-api-access-mg8xs\" (UniqueName: \"kubernetes.io/projected/514b08d9-34ac-4867-90a4-53fd518d25c1-kube-api-access-mg8xs\") pod \"514b08d9-34ac-4867-90a4-53fd518d25c1\" (UID: \"514b08d9-34ac-4867-90a4-53fd518d25c1\") " Nov 04 01:26:13.538004 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.537978 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514b08d9-34ac-4867-90a4-53fd518d25c1-plugins-config-volume" pod "514b08d9-34ac-4867-90a4-53fd518d25c1" (UID: "514b08d9-34ac-4867-90a4-53fd518d25c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:13.538347 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.538326 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glx77\" (UniqueName: \"kubernetes.io/projected/e87443b1-c333-4d70-9843-e604b060f596-kube-api-access-glx77\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:13.538450 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.538353 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e87443b1-c333-4d70-9843-e604b060f596-maas-api-tls\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:13.538450 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.538367 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514b08d9-34ac-4867-90a4-53fd518d25c1-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:13.538450 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.538381 4852 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e87443b1-c333-4d70-9843-e604b060f596-metrics-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:13.538878 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.538846 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc338560-6a2d-4fdd-bd5c-23e1071d63a7-plugins-config-volume" pod "dc338560-6a2d-4fdd-bd5c-23e1071d63a7" (UID: "dc338560-6a2d-4fdd-bd5c-23e1071d63a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:13.540592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.540525 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc338560-6a2d-4fdd-bd5c-23e1071d63a7-kube-api-access-vv2rl" pod "dc338560-6a2d-4fdd-bd5c-23e1071d63a7" (UID: "dc338560-6a2d-4fdd-bd5c-23e1071d63a7"). InnerVolumeSpecName "kube-api-access-vv2rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:13.540592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.540545 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514b08d9-34ac-4867-90a4-53fd518d25c1-kube-api-access-mg8xs" pod "514b08d9-34ac-4867-90a4-53fd518d25c1" (UID: "514b08d9-34ac-4867-90a4-53fd518d25c1"). InnerVolumeSpecName "kube-api-access-mg8xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:13.541600 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.541534 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb"] Nov 04 01:26:13.542613 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.542593 4852 scope.go:122] "RemoveContainer" containerID="c34f41912208cbe8d8f260a26b17dc41f6595585a46501d9996756692052c104" Nov 04 01:26:13.542938 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:26:13.542913 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c34f41912208cbe8d8f260a26b17dc41f6595585a46501d9996756692052c104\": container with ID starting with c34f41912208cbe8d8f260a26b17dc41f6595585a46501d9996756692052c104 not found: ID does not exist" containerID="c34f41912208cbe8d8f260a26b17dc41f6595585a46501d9996756692052c104" Nov 04 01:26:13.543019 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.542946 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c34f41912208cbe8d8f260a26b17dc41f6595585a46501d9996756692052c104"} err="failed to get container status \"c34f41912208cbe8d8f260a26b17dc41f6595585a46501d9996756692052c104\": rpc error: code = NotFound desc = could not find container \"c34f41912208cbe8d8f260a26b17dc41f6595585a46501d9996756692052c104\": container with ID starting with c34f41912208cbe8d8f260a26b17dc41f6595585a46501d9996756692052c104 not found: ID does not exist" Nov 04 01:26:13.548241 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.548219 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-296c124f-f68689bf6-qg9mb"] Nov 04 01:26:13.639304 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.639280 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv2rl\" (UniqueName: \"kubernetes.io/projected/dc338560-6a2d-4fdd-bd5c-23e1071d63a7-kube-api-access-vv2rl\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:13.639635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.639307 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc338560-6a2d-4fdd-bd5c-23e1071d63a7-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:13.639635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.639323 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg8xs\" (UniqueName: \"kubernetes.io/projected/514b08d9-34ac-4867-90a4-53fd518d25c1-kube-api-access-mg8xs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:13.799282 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.799258 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf"] Nov 04 01:26:13.807071 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.807052 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-296c124f-8b4b65755-qn6cf"] Nov 04 01:26:13.822784 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.822764 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2"] Nov 04 01:26:13.833823 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.833802 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-296c124f-7bb687db96-25vb2"] Nov 04 01:26:13.917162 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.916361 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:13.995162 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.995130 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="514b08d9-34ac-4867-90a4-53fd518d25c1" path="/var/lib/kubelet/pods/514b08d9-34ac-4867-90a4-53fd518d25c1/volumes" Nov 04 01:26:13.995633 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.995618 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc338560-6a2d-4fdd-bd5c-23e1071d63a7" path="/var/lib/kubelet/pods/dc338560-6a2d-4fdd-bd5c-23e1071d63a7/volumes" Nov 04 01:26:13.996035 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:13.996023 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e87443b1-c333-4d70-9843-e604b060f596" path="/var/lib/kubelet/pods/e87443b1-c333-4d70-9843-e604b060f596/volumes" Nov 04 01:26:14.584006 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:14.583976 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:15.080604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.078131 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:15.401490 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.401409 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:15.655963 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.655867 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl"] Nov 04 01:26:15.656747 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.656721 4852 cpu_manager.go:414] "containerMap: removing container" podUID="514b08d9-34ac-4867-90a4-53fd518d25c1" containerName="payload-processing" Nov 04 01:26:15.656747 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.656747 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="514b08d9-34ac-4867-90a4-53fd518d25c1" containerName="payload-processing" Nov 04 01:26:15.656952 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.656763 4852 cpu_manager.go:414] "containerMap: removing container" podUID="e87443b1-c333-4d70-9843-e604b060f596" containerName="maas-api" Nov 04 01:26:15.656952 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.656771 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e87443b1-c333-4d70-9843-e604b060f596" containerName="maas-api" Nov 04 01:26:15.656952 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.656832 4852 cpu_manager.go:414] "containerMap: removing container" podUID="eef87d33-f22e-4ce2-ad38-94ac8f23913f" containerName="istio-proxy" Nov 04 01:26:15.656952 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.656840 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eef87d33-f22e-4ce2-ad38-94ac8f23913f" containerName="istio-proxy" Nov 04 01:26:15.656952 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.656850 4852 cpu_manager.go:414] "containerMap: removing container" podUID="5972b292-b0b7-45af-9674-f0c00f20c1c9" containerName="maas-api" Nov 04 01:26:15.656952 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.656858 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5972b292-b0b7-45af-9674-f0c00f20c1c9" containerName="maas-api" Nov 04 01:26:15.656952 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.656868 4852 cpu_manager.go:414] "containerMap: removing container" podUID="dc338560-6a2d-4fdd-bd5c-23e1071d63a7" containerName="payload-pre-processing" Nov 04 01:26:15.656952 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.656876 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc338560-6a2d-4fdd-bd5c-23e1071d63a7" containerName="payload-pre-processing" Nov 04 01:26:15.657353 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.657015 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="514b08d9-34ac-4867-90a4-53fd518d25c1" containerName="payload-processing" Nov 04 01:26:15.657353 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.657029 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="e87443b1-c333-4d70-9843-e604b060f596" containerName="maas-api" Nov 04 01:26:15.657353 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.657048 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="eef87d33-f22e-4ce2-ad38-94ac8f23913f" containerName="istio-proxy" Nov 04 01:26:15.657353 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.657057 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="5972b292-b0b7-45af-9674-f0c00f20c1c9" containerName="maas-api" Nov 04 01:26:15.657353 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.657065 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc338560-6a2d-4fdd-bd5c-23e1071d63a7" containerName="payload-pre-processing" Nov 04 01:26:15.667306 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.667279 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.681133 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.681108 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-dfcebf-openshift-default-dockercfg-qnv2j\"" Nov 04 01:26:15.684077 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.684050 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl"] Nov 04 01:26:15.722021 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.721985 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-token\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.722185 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.722052 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-credential-socket\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.722259 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.722219 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhqw7\" (UniqueName: \"kubernetes.io/projected/9febb536-9d73-41c8-8a6e-5b2aa1446825-kube-api-access-mhqw7\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.722318 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.722298 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-podinfo\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.722380 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.722364 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-data\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.722435 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.722403 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-envoy\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.722515 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.722492 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9febb536-9d73-41c8-8a6e-5b2aa1446825-istiod-ca-cert\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.722631 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.722617 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-workload-certs\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.722693 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.722650 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-workload-socket\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.799213 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.794902 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:15.826517 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.826468 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-token\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.876719 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.876682 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-credential-socket\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.877071 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.877044 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhqw7\" (UniqueName: \"kubernetes.io/projected/9febb536-9d73-41c8-8a6e-5b2aa1446825-kube-api-access-mhqw7\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.880720 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.880689 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-podinfo\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.880957 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.880939 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-data\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.881111 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.881090 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-envoy\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.887345 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.886637 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9febb536-9d73-41c8-8a6e-5b2aa1446825-istiod-ca-cert\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.887626 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.884307 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-data\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.887626 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.879488 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-credential-socket\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.887626 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.887305 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9febb536-9d73-41c8-8a6e-5b2aa1446825-istiod-ca-cert\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.894769 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.894746 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-workload-certs\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.894915 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.894900 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-workload-socket\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.902586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.895426 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-workload-socket\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.907624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.895895 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-workload-certs\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.950587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.927719 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-token\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.950587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.930809 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-envoy\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.950587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.939157 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-podinfo\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:15.950587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:15.942134 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhqw7\" (UniqueName: \"kubernetes.io/projected/9febb536-9d73-41c8-8a6e-5b2aa1446825-kube-api-access-mhqw7\") pod \"e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:16.029454 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:16.029172 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:16.429675 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:16.426431 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:16.454517 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:26:16.454478 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9febb536_9d73_41c8_8a6e_5b2aa1446825.slice/crio-6c310d2d5cd9c10dbc64e197a0a00afce52faebe048fc593dc5be04a068ecedc WatchSource:0}: Error finding container 6c310d2d5cd9c10dbc64e197a0a00afce52faebe048fc593dc5be04a068ecedc: Status 404 returned error can't find the container with id 6c310d2d5cd9c10dbc64e197a0a00afce52faebe048fc593dc5be04a068ecedc Nov 04 01:26:16.455003 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:16.454970 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl"] Nov 04 01:26:16.457984 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:16.457932 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:26:16.458079 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:16.458042 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:26:16.458122 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:16.458089 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:26:16.534481 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:16.534450 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" event={"ID":"9febb536-9d73-41c8-8a6e-5b2aa1446825","Type":"ContainerStarted","Data":"6c310d2d5cd9c10dbc64e197a0a00afce52faebe048fc593dc5be04a068ecedc"} Nov 04 01:26:17.344866 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:17.344830 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:17.542470 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:17.542435 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" event={"ID":"9febb536-9d73-41c8-8a6e-5b2aa1446825","Type":"ContainerStarted","Data":"1fe10790ad7324f595f5dc7fc778f0fce0aafce8523825cab82e73850efb7e85"} Nov 04 01:26:17.580504 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:17.580443 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" podStartSLOduration=2.580424447 podStartE2EDuration="2.580424447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:17.575194552 +0000 UTC m=+1659.690304696" watchObservedRunningTime="2026-11-04 01:26:17.580424447 +0000 UTC m=+1659.695534567" Nov 04 01:26:18.032772 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:18.032736 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:18.039463 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:18.039429 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:18.163057 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:18.162918 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:18.491420 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:18.491245 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:18.551595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:18.551470 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:18.552546 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:18.552522 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:26:19.712646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:19.712587 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:20.020481 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:20.020451 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:20.190409 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:20.190375 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:20.769086 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:20.768982 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:21.288739 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.288705 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:21.319899 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.319872 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv"] Nov 04 01:26:21.323792 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.323774 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv" Nov 04 01:26:21.334937 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.334914 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv"] Nov 04 01:26:21.338627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.338604 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-dfcebf\"" Nov 04 01:26:21.338749 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.338673 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-dfcebf-dockercfg-x9xgx\"" Nov 04 01:26:21.361183 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.361153 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e92e697-d5a8-4f9c-9716-697d3f9e2546-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv\" (UID: \"0e92e697-d5a8-4f9c-9716-697d3f9e2546\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv" Nov 04 01:26:21.361320 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.361194 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cddwc\" (UniqueName: \"kubernetes.io/projected/0e92e697-d5a8-4f9c-9716-697d3f9e2546-kube-api-access-cddwc\") pod \"payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv\" (UID: \"0e92e697-d5a8-4f9c-9716-697d3f9e2546\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv" Nov 04 01:26:21.369798 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.369770 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g"] Nov 04 01:26:21.373804 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.373787 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g" Nov 04 01:26:21.391913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.391889 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g"] Nov 04 01:26:21.487018 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.463595 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhfdw\" (UniqueName: \"kubernetes.io/projected/39133c56-c8e4-4b42-9df6-7dc091b4f09d-kube-api-access-rhfdw\") pod \"payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g\" (UID: \"39133c56-c8e4-4b42-9df6-7dc091b4f09d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g" Nov 04 01:26:21.487939 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.487900 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e92e697-d5a8-4f9c-9716-697d3f9e2546-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv\" (UID: \"0e92e697-d5a8-4f9c-9716-697d3f9e2546\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv" Nov 04 01:26:21.492629 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.489642 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cddwc\" (UniqueName: \"kubernetes.io/projected/0e92e697-d5a8-4f9c-9716-697d3f9e2546-kube-api-access-cddwc\") pod \"payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv\" (UID: \"0e92e697-d5a8-4f9c-9716-697d3f9e2546\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv" Nov 04 01:26:21.492629 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.489903 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39133c56-c8e4-4b42-9df6-7dc091b4f09d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g\" (UID: \"39133c56-c8e4-4b42-9df6-7dc091b4f09d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g" Nov 04 01:26:21.492629 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.491068 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e92e697-d5a8-4f9c-9716-697d3f9e2546-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv\" (UID: \"0e92e697-d5a8-4f9c-9716-697d3f9e2546\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv" Nov 04 01:26:21.554640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.553129 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:21.591659 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.591209 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39133c56-c8e4-4b42-9df6-7dc091b4f09d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g\" (UID: \"39133c56-c8e4-4b42-9df6-7dc091b4f09d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g" Nov 04 01:26:21.591659 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.591406 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhfdw\" (UniqueName: \"kubernetes.io/projected/39133c56-c8e4-4b42-9df6-7dc091b4f09d-kube-api-access-rhfdw\") pod \"payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g\" (UID: \"39133c56-c8e4-4b42-9df6-7dc091b4f09d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g" Nov 04 01:26:21.593200 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.593134 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39133c56-c8e4-4b42-9df6-7dc091b4f09d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g\" (UID: \"39133c56-c8e4-4b42-9df6-7dc091b4f09d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g" Nov 04 01:26:21.640019 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.639988 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cddwc\" (UniqueName: \"kubernetes.io/projected/0e92e697-d5a8-4f9c-9716-697d3f9e2546-kube-api-access-cddwc\") pod \"payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv\" (UID: \"0e92e697-d5a8-4f9c-9716-697d3f9e2546\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv" Nov 04 01:26:21.640466 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.640443 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv" Nov 04 01:26:21.641638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.641615 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhfdw\" (UniqueName: \"kubernetes.io/projected/39133c56-c8e4-4b42-9df6-7dc091b4f09d-kube-api-access-rhfdw\") pod \"payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g\" (UID: \"39133c56-c8e4-4b42-9df6-7dc091b4f09d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g" Nov 04 01:26:21.697750 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.697720 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g" Nov 04 01:26:21.886807 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:21.883546 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:22.067894 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:22.067863 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g"] Nov 04 01:26:22.069597 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:26:22.069557 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39133c56_c8e4_4b42_9df6_7dc091b4f09d.slice/crio-6f252deab79ed469903186cf4a3dd87f3c8185062080c9098ec326c258f012a9 WatchSource:0}: Error finding container 6f252deab79ed469903186cf4a3dd87f3c8185062080c9098ec326c258f012a9: Status 404 returned error can't find the container with id 6f252deab79ed469903186cf4a3dd87f3c8185062080c9098ec326c258f012a9 Nov 04 01:26:22.086451 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:22.086345 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv"] Nov 04 01:26:22.609898 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:22.609825 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:22.694586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:22.685375 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv" event={"ID":"0e92e697-d5a8-4f9c-9716-697d3f9e2546","Type":"ContainerStarted","Data":"4e9c4129112966b94dde1fb82a86d10a71a295309b5ee5d3731f5bf441387952"} Nov 04 01:26:22.694586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:22.685441 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv" event={"ID":"0e92e697-d5a8-4f9c-9716-697d3f9e2546","Type":"ContainerStarted","Data":"62a57276ed7106e62743db74206ecc11da972fee8ae045eeacb87a2358124530"} Nov 04 01:26:22.694586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:22.686804 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv" Nov 04 01:26:22.694586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:22.692105 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g" event={"ID":"39133c56-c8e4-4b42-9df6-7dc091b4f09d","Type":"ContainerStarted","Data":"ce0a204c3e522d9d0e14fcc69b0bdb1f08dd639cf3b142b5d3428d0e795ae197"} Nov 04 01:26:22.694586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:22.692162 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g" event={"ID":"39133c56-c8e4-4b42-9df6-7dc091b4f09d","Type":"ContainerStarted","Data":"6f252deab79ed469903186cf4a3dd87f3c8185062080c9098ec326c258f012a9"} Nov 04 01:26:22.694586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:22.694156 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g" Nov 04 01:26:22.719603 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:22.711473 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv" podStartSLOduration=1.71145379 podStartE2EDuration="1.71145379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:22.710125504 +0000 UTC m=+1664.825235624" watchObservedRunningTime="2026-11-04 01:26:22.71145379 +0000 UTC m=+1664.826563908" Nov 04 01:26:22.731510 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:22.731462 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g" podStartSLOduration=1.73144879 podStartE2EDuration="1.73144879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:22.728417801 +0000 UTC m=+1664.843527920" watchObservedRunningTime="2026-11-04 01:26:22.73144879 +0000 UTC m=+1664.846558908" Nov 04 01:26:23.436587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:23.436089 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:24.201595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:24.200442 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:24.909538 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:24.909507 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:25.650226 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:25.650192 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:25.960903 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:25.960815 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb"] Nov 04 01:26:25.971750 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:25.971724 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q"] Nov 04 01:26:26.091946 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:26.091910 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:26.259173 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:26.259129 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:26.420070 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:26.420035 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:27.295618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:27.295581 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:27.467727 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:27.467694 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:28.357397 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:28.357365 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:28.854801 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:28.854755 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk"] Nov 04 01:26:28.855025 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:28.854994 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" podUID="70aa5fc4-7c6f-4748-85a0-091d29781f8c" containerName="maas-api" containerID="cri-o://3fd0608c52d2e8dcba4ba26642846ba56cae585755971659d4a5c08bb6a68181" gracePeriod=30 Nov 04 01:26:28.932380 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:28.932339 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" podUID="70aa5fc4-7c6f-4748-85a0-091d29781f8c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.80:8443/health\": dial tcp 10.134.0.80:8443: connect: connection refused" Nov 04 01:26:28.954676 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:28.954641 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:29.132201 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.132125 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:29.153415 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.153390 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" Nov 04 01:26:29.296919 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.296891 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70aa5fc4-7c6f-4748-85a0-091d29781f8c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70aa5fc4-7c6f-4748-85a0-091d29781f8c-metrics-certs\") pod \"70aa5fc4-7c6f-4748-85a0-091d29781f8c\" (UID: \"70aa5fc4-7c6f-4748-85a0-091d29781f8c\") " Nov 04 01:26:29.297124 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.296998 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70aa5fc4-7c6f-4748-85a0-091d29781f8c-kube-api-access-vmjmq\" (UniqueName: \"kubernetes.io/projected/70aa5fc4-7c6f-4748-85a0-091d29781f8c-kube-api-access-vmjmq\") pod \"70aa5fc4-7c6f-4748-85a0-091d29781f8c\" (UID: \"70aa5fc4-7c6f-4748-85a0-091d29781f8c\") " Nov 04 01:26:29.297124 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.297044 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70aa5fc4-7c6f-4748-85a0-091d29781f8c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70aa5fc4-7c6f-4748-85a0-091d29781f8c-maas-api-tls\") pod \"70aa5fc4-7c6f-4748-85a0-091d29781f8c\" (UID: \"70aa5fc4-7c6f-4748-85a0-091d29781f8c\") " Nov 04 01:26:29.297124 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.297119 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70aa5fc4-7c6f-4748-85a0-091d29781f8c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70aa5fc4-7c6f-4748-85a0-091d29781f8c-openshift-service-ca\") pod \"70aa5fc4-7c6f-4748-85a0-091d29781f8c\" (UID: \"70aa5fc4-7c6f-4748-85a0-091d29781f8c\") " Nov 04 01:26:29.297557 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.297518 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70aa5fc4-7c6f-4748-85a0-091d29781f8c-openshift-service-ca" pod "70aa5fc4-7c6f-4748-85a0-091d29781f8c" (UID: "70aa5fc4-7c6f-4748-85a0-091d29781f8c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:29.297874 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.297754 4852 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70aa5fc4-7c6f-4748-85a0-091d29781f8c-openshift-service-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:29.299269 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.299238 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70aa5fc4-7c6f-4748-85a0-091d29781f8c-kube-api-access-vmjmq" pod "70aa5fc4-7c6f-4748-85a0-091d29781f8c" (UID: "70aa5fc4-7c6f-4748-85a0-091d29781f8c"). InnerVolumeSpecName "kube-api-access-vmjmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:29.299833 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.299806 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70aa5fc4-7c6f-4748-85a0-091d29781f8c-metrics-certs" pod "70aa5fc4-7c6f-4748-85a0-091d29781f8c" (UID: "70aa5fc4-7c6f-4748-85a0-091d29781f8c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:26:29.299907 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.299885 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70aa5fc4-7c6f-4748-85a0-091d29781f8c-maas-api-tls" pod "70aa5fc4-7c6f-4748-85a0-091d29781f8c" (UID: "70aa5fc4-7c6f-4748-85a0-091d29781f8c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:26:29.304813 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.304791 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:29.399471 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.399386 4852 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70aa5fc4-7c6f-4748-85a0-091d29781f8c-metrics-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:29.399471 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.399418 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmjmq\" (UniqueName: \"kubernetes.io/projected/70aa5fc4-7c6f-4748-85a0-091d29781f8c-kube-api-access-vmjmq\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:29.399471 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.399433 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70aa5fc4-7c6f-4748-85a0-091d29781f8c-maas-api-tls\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:29.640397 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.640333 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:29.791004 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.790968 4852 generic.go:356] "Generic (PLEG): container finished" podID="70aa5fc4-7c6f-4748-85a0-091d29781f8c" containerID="3fd0608c52d2e8dcba4ba26642846ba56cae585755971659d4a5c08bb6a68181" exitCode=0 Nov 04 01:26:29.791444 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.791408 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" event={"ID":"70aa5fc4-7c6f-4748-85a0-091d29781f8c","Type":"ContainerDied","Data":"3fd0608c52d2e8dcba4ba26642846ba56cae585755971659d4a5c08bb6a68181"} Nov 04 01:26:29.791616 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.791511 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" Nov 04 01:26:29.791616 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.791548 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk" event={"ID":"70aa5fc4-7c6f-4748-85a0-091d29781f8c","Type":"ContainerDied","Data":"3400e243c96b0d0899f03a63cc5f49c219210b300bd65653574518149f710912"} Nov 04 01:26:29.791753 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.791636 4852 scope.go:122] "RemoveContainer" containerID="3fd0608c52d2e8dcba4ba26642846ba56cae585755971659d4a5c08bb6a68181" Nov 04 01:26:29.807824 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.807806 4852 scope.go:122] "RemoveContainer" containerID="3fd0608c52d2e8dcba4ba26642846ba56cae585755971659d4a5c08bb6a68181" Nov 04 01:26:29.808105 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:26:29.808087 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fd0608c52d2e8dcba4ba26642846ba56cae585755971659d4a5c08bb6a68181\": container with ID starting with 3fd0608c52d2e8dcba4ba26642846ba56cae585755971659d4a5c08bb6a68181 not found: ID does not exist" containerID="3fd0608c52d2e8dcba4ba26642846ba56cae585755971659d4a5c08bb6a68181" Nov 04 01:26:29.808186 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.808118 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fd0608c52d2e8dcba4ba26642846ba56cae585755971659d4a5c08bb6a68181"} err="failed to get container status \"3fd0608c52d2e8dcba4ba26642846ba56cae585755971659d4a5c08bb6a68181\": rpc error: code = NotFound desc = could not find container \"3fd0608c52d2e8dcba4ba26642846ba56cae585755971659d4a5c08bb6a68181\": container with ID starting with 3fd0608c52d2e8dcba4ba26642846ba56cae585755971659d4a5c08bb6a68181 not found: ID does not exist" Nov 04 01:26:29.824589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.824554 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk"] Nov 04 01:26:29.832884 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.832864 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-62510d3b-68b5fcff4f-p2khk"] Nov 04 01:26:29.868579 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:26:29.868521 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:26:29.995076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:29.995043 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70aa5fc4-7c6f-4748-85a0-091d29781f8c" path="/var/lib/kubelet/pods/70aa5fc4-7c6f-4748-85a0-091d29781f8c/volumes" Nov 04 01:26:30.325838 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.325803 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:30.493369 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.493340 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:30.633922 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.633849 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f"] Nov 04 01:26:30.634821 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.634798 4852 cpu_manager.go:414] "containerMap: removing container" podUID="70aa5fc4-7c6f-4748-85a0-091d29781f8c" containerName="maas-api" Nov 04 01:26:30.634946 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.634824 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70aa5fc4-7c6f-4748-85a0-091d29781f8c" containerName="maas-api" Nov 04 01:26:30.635033 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.635018 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="70aa5fc4-7c6f-4748-85a0-091d29781f8c" containerName="maas-api" Nov 04 01:26:30.640630 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.640605 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.648366 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.648342 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-af6909-openshift-default-dockercfg-qqsxm\"" Nov 04 01:26:30.652091 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.652068 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f"] Nov 04 01:26:30.737837 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.737793 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31799de-2b0d-4f2e-839e-058668f304c6-istio-podinfo\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.738040 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.737866 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31799de-2b0d-4f2e-839e-058668f304c6-istio-token\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.738040 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.737932 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-istio-data\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.738040 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.737973 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-workload-certs\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.738040 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.738019 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-istio-envoy\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.738255 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.738043 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-credential-socket\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.738255 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.738142 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4rzv\" (UniqueName: \"kubernetes.io/projected/d31799de-2b0d-4f2e-839e-058668f304c6-kube-api-access-p4rzv\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.738255 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.738184 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31799de-2b0d-4f2e-839e-058668f304c6-istiod-ca-cert\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.738407 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.738264 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-workload-socket\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.809588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.808958 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:30.839495 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.839460 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31799de-2b0d-4f2e-839e-058668f304c6-istio-podinfo\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.839796 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.839774 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31799de-2b0d-4f2e-839e-058668f304c6-istio-token\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.839960 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.839945 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-istio-data\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.840083 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.840069 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-workload-certs\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.840205 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.840192 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-istio-envoy\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.840309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.840296 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-credential-socket\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.840482 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.840468 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4rzv\" (UniqueName: \"kubernetes.io/projected/d31799de-2b0d-4f2e-839e-058668f304c6-kube-api-access-p4rzv\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.840629 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.840614 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31799de-2b0d-4f2e-839e-058668f304c6-istiod-ca-cert\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.840803 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.840789 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-workload-socket\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.841414 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.841391 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-workload-socket\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.842809 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.842789 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-credential-socket\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.850999 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.843228 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-istio-data\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.876839 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.849827 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-workload-certs\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.877022 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.850943 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31799de-2b0d-4f2e-839e-058668f304c6-istiod-ca-cert\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.877158 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.857068 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31799de-2b0d-4f2e-839e-058668f304c6-istio-podinfo\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.877252 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.875850 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-istio-envoy\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.880585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.878027 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31799de-2b0d-4f2e-839e-058668f304c6-istio-token\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.893146 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.892684 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4rzv\" (UniqueName: \"kubernetes.io/projected/d31799de-2b0d-4f2e-839e-058668f304c6-kube-api-access-p4rzv\") pod \"e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.963040 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.962780 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:30.963468 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.963283 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb" podUID="3ea5f3c5-a60c-4781-87de-89edbb87c02a" containerName="payload-pre-processing" containerID="cri-o://177f3d8608fac8cd5e5d126369a47adfbe50ba73bba9404ae9093e73b992476e" gracePeriod=25 Nov 04 01:26:30.983449 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:30.982475 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q" podUID="ce052f4b-e92c-4795-a4ba-947cf1c824cf" containerName="payload-processing" containerID="cri-o://1b11c9560f937fe974a749feab5f9a98e795c9e9b07b8feab6eac0ae31c1afd0" gracePeriod=25 Nov 04 01:26:31.047752 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.046588 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:31.255892 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.255619 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:31.457177 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.457082 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f"] Nov 04 01:26:31.469599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.462206 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:26:31.469599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.462291 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:26:31.469599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.462340 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:26:31.597389 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.597364 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q" Nov 04 01:26:31.634420 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.634389 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce052f4b-e92c-4795-a4ba-947cf1c824cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce052f4b-e92c-4795-a4ba-947cf1c824cf-plugins-config-volume\") pod \"ce052f4b-e92c-4795-a4ba-947cf1c824cf\" (UID: \"ce052f4b-e92c-4795-a4ba-947cf1c824cf\") " Nov 04 01:26:31.634627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.634605 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce052f4b-e92c-4795-a4ba-947cf1c824cf-kube-api-access-xqrcg\" (UniqueName: \"kubernetes.io/projected/ce052f4b-e92c-4795-a4ba-947cf1c824cf-kube-api-access-xqrcg\") pod \"ce052f4b-e92c-4795-a4ba-947cf1c824cf\" (UID: \"ce052f4b-e92c-4795-a4ba-947cf1c824cf\") " Nov 04 01:26:31.636136 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.636106 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce052f4b-e92c-4795-a4ba-947cf1c824cf-plugins-config-volume" pod "ce052f4b-e92c-4795-a4ba-947cf1c824cf" (UID: "ce052f4b-e92c-4795-a4ba-947cf1c824cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:31.637784 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.637765 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce052f4b-e92c-4795-a4ba-947cf1c824cf-kube-api-access-xqrcg" pod "ce052f4b-e92c-4795-a4ba-947cf1c824cf" (UID: "ce052f4b-e92c-4795-a4ba-947cf1c824cf"). InnerVolumeSpecName "kube-api-access-xqrcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:31.669916 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.669889 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb" Nov 04 01:26:31.737269 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.737179 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ea5f3c5-a60c-4781-87de-89edbb87c02a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea5f3c5-a60c-4781-87de-89edbb87c02a-plugins-config-volume\") pod \"3ea5f3c5-a60c-4781-87de-89edbb87c02a\" (UID: \"3ea5f3c5-a60c-4781-87de-89edbb87c02a\") " Nov 04 01:26:31.737651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.737625 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea5f3c5-a60c-4781-87de-89edbb87c02a-kube-api-access-779tg\" (UniqueName: \"kubernetes.io/projected/3ea5f3c5-a60c-4781-87de-89edbb87c02a-kube-api-access-779tg\") pod \"3ea5f3c5-a60c-4781-87de-89edbb87c02a\" (UID: \"3ea5f3c5-a60c-4781-87de-89edbb87c02a\") " Nov 04 01:26:31.738265 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.738239 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea5f3c5-a60c-4781-87de-89edbb87c02a-plugins-config-volume" pod "3ea5f3c5-a60c-4781-87de-89edbb87c02a" (UID: "3ea5f3c5-a60c-4781-87de-89edbb87c02a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:31.738666 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.738385 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce052f4b-e92c-4795-a4ba-947cf1c824cf-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:31.738762 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.738694 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqrcg\" (UniqueName: \"kubernetes.io/projected/ce052f4b-e92c-4795-a4ba-947cf1c824cf-kube-api-access-xqrcg\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:31.741512 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.741488 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea5f3c5-a60c-4781-87de-89edbb87c02a-kube-api-access-779tg" pod "3ea5f3c5-a60c-4781-87de-89edbb87c02a" (UID: "3ea5f3c5-a60c-4781-87de-89edbb87c02a"). InnerVolumeSpecName "kube-api-access-779tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:31.840062 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.840026 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea5f3c5-a60c-4781-87de-89edbb87c02a-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:31.840062 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.840065 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-779tg\" (UniqueName: \"kubernetes.io/projected/3ea5f3c5-a60c-4781-87de-89edbb87c02a-kube-api-access-779tg\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:31.841168 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.841134 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" event={"ID":"d31799de-2b0d-4f2e-839e-058668f304c6","Type":"ContainerStarted","Data":"09ce9b824a363adf76899b88511527a5e5b7bed4d59032c212dafa11238668f7"} Nov 04 01:26:31.841315 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.841196 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" event={"ID":"d31799de-2b0d-4f2e-839e-058668f304c6","Type":"ContainerStarted","Data":"c5f08847de7e1f272ed61e5b9e500e15a6e0a4c8fbdbea5252d15ec952b3d07b"} Nov 04 01:26:31.842793 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.842762 4852 generic.go:356] "Generic (PLEG): container finished" podID="3ea5f3c5-a60c-4781-87de-89edbb87c02a" containerID="177f3d8608fac8cd5e5d126369a47adfbe50ba73bba9404ae9093e73b992476e" exitCode=0 Nov 04 01:26:31.842908 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.842828 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb" Nov 04 01:26:31.842908 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.842836 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb" event={"ID":"3ea5f3c5-a60c-4781-87de-89edbb87c02a","Type":"ContainerDied","Data":"177f3d8608fac8cd5e5d126369a47adfbe50ba73bba9404ae9093e73b992476e"} Nov 04 01:26:31.843024 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.842917 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb" event={"ID":"3ea5f3c5-a60c-4781-87de-89edbb87c02a","Type":"ContainerDied","Data":"0ba87f9bbd8dcf41924c4ba341798563db1e13a9279a7f8683a50c97da2b240e"} Nov 04 01:26:31.843024 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.842951 4852 scope.go:122] "RemoveContainer" containerID="177f3d8608fac8cd5e5d126369a47adfbe50ba73bba9404ae9093e73b992476e" Nov 04 01:26:31.850812 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.850790 4852 generic.go:356] "Generic (PLEG): container finished" podID="ce052f4b-e92c-4795-a4ba-947cf1c824cf" containerID="1b11c9560f937fe974a749feab5f9a98e795c9e9b07b8feab6eac0ae31c1afd0" exitCode=0 Nov 04 01:26:31.850917 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.850837 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q" Nov 04 01:26:31.850917 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.850845 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q" event={"ID":"ce052f4b-e92c-4795-a4ba-947cf1c824cf","Type":"ContainerDied","Data":"1b11c9560f937fe974a749feab5f9a98e795c9e9b07b8feab6eac0ae31c1afd0"} Nov 04 01:26:31.850917 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.850885 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q" event={"ID":"ce052f4b-e92c-4795-a4ba-947cf1c824cf","Type":"ContainerDied","Data":"0c500289540304592268f997aefc1b370997e6dca7877eac9cfd060fbfce50c3"} Nov 04 01:26:31.870946 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.869222 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" podStartSLOduration=1.869206205 podStartE2EDuration="1.869206205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:31.863317062 +0000 UTC m=+1673.978427182" watchObservedRunningTime="2026-11-04 01:26:31.869206205 +0000 UTC m=+1673.984316324" Nov 04 01:26:31.886430 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.886415 4852 scope.go:122] "RemoveContainer" containerID="177f3d8608fac8cd5e5d126369a47adfbe50ba73bba9404ae9093e73b992476e" Nov 04 01:26:31.886881 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:26:31.886860 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"177f3d8608fac8cd5e5d126369a47adfbe50ba73bba9404ae9093e73b992476e\": container with ID starting with 177f3d8608fac8cd5e5d126369a47adfbe50ba73bba9404ae9093e73b992476e not found: ID does not exist" containerID="177f3d8608fac8cd5e5d126369a47adfbe50ba73bba9404ae9093e73b992476e" Nov 04 01:26:31.886966 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.886886 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177f3d8608fac8cd5e5d126369a47adfbe50ba73bba9404ae9093e73b992476e"} err="failed to get container status \"177f3d8608fac8cd5e5d126369a47adfbe50ba73bba9404ae9093e73b992476e\": rpc error: code = NotFound desc = could not find container \"177f3d8608fac8cd5e5d126369a47adfbe50ba73bba9404ae9093e73b992476e\": container with ID starting with 177f3d8608fac8cd5e5d126369a47adfbe50ba73bba9404ae9093e73b992476e not found: ID does not exist" Nov 04 01:26:31.886966 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.886903 4852 scope.go:122] "RemoveContainer" containerID="1b11c9560f937fe974a749feab5f9a98e795c9e9b07b8feab6eac0ae31c1afd0" Nov 04 01:26:31.898851 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.898828 4852 scope.go:122] "RemoveContainer" containerID="1b11c9560f937fe974a749feab5f9a98e795c9e9b07b8feab6eac0ae31c1afd0" Nov 04 01:26:31.899138 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:26:31.899119 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b11c9560f937fe974a749feab5f9a98e795c9e9b07b8feab6eac0ae31c1afd0\": container with ID starting with 1b11c9560f937fe974a749feab5f9a98e795c9e9b07b8feab6eac0ae31c1afd0 not found: ID does not exist" containerID="1b11c9560f937fe974a749feab5f9a98e795c9e9b07b8feab6eac0ae31c1afd0" Nov 04 01:26:31.899190 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.899144 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b11c9560f937fe974a749feab5f9a98e795c9e9b07b8feab6eac0ae31c1afd0"} err="failed to get container status \"1b11c9560f937fe974a749feab5f9a98e795c9e9b07b8feab6eac0ae31c1afd0\": rpc error: code = NotFound desc = could not find container \"1b11c9560f937fe974a749feab5f9a98e795c9e9b07b8feab6eac0ae31c1afd0\": container with ID starting with 1b11c9560f937fe974a749feab5f9a98e795c9e9b07b8feab6eac0ae31c1afd0 not found: ID does not exist" Nov 04 01:26:31.907356 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.907335 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q"] Nov 04 01:26:31.914674 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.914655 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-62510d3b-fd9fc56b-68r7q"] Nov 04 01:26:31.930047 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.930021 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb"] Nov 04 01:26:31.932988 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.932964 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb"] Nov 04 01:26:31.964678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.964654 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:31.998909 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.998831 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ea5f3c5-a60c-4781-87de-89edbb87c02a" path="/var/lib/kubelet/pods/3ea5f3c5-a60c-4781-87de-89edbb87c02a/volumes" Nov 04 01:26:31.999292 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:31.999277 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce052f4b-e92c-4795-a4ba-947cf1c824cf" path="/var/lib/kubelet/pods/ce052f4b-e92c-4795-a4ba-947cf1c824cf/volumes" Nov 04 01:26:32.126758 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:32.126703 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:32.300046 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:32.299998 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:32.636439 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:32.636338 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-derive-62510d3b-54ddf77599-nvcvb" podUID="3ea5f3c5-a60c-4781-87de-89edbb87c02a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.77:50052\" within 1s: context deadline exceeded" Nov 04 01:26:32.636439 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:26:32.636393 4852 logging.go:55] [core] [Channel #434 SubChannel #435]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.77:50052", ServerName: "10.134.0.77:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.77:50052: operation was canceled" Nov 04 01:26:32.838104 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:32.838067 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:32.969901 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:32.969823 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:32.999924 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:32.999892 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:33.569124 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:33.569090 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:33.766139 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:33.764521 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:33.910120 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:33.910042 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:33.910907 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:33.910892 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:26:34.440851 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:34.440818 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:34.640095 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:34.640069 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:34.708548 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:34.708463 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv" Nov 04 01:26:34.708724 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:34.708683 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g" Nov 04 01:26:34.786702 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:34.786677 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:35.527011 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.526985 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:35.613545 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.613517 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb"] Nov 04 01:26:35.614074 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.614061 4852 cpu_manager.go:414] "containerMap: removing container" podUID="3ea5f3c5-a60c-4781-87de-89edbb87c02a" containerName="payload-pre-processing" Nov 04 01:26:35.614118 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.614076 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea5f3c5-a60c-4781-87de-89edbb87c02a" containerName="payload-pre-processing" Nov 04 01:26:35.614118 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.614102 4852 cpu_manager.go:414] "containerMap: removing container" podUID="ce052f4b-e92c-4795-a4ba-947cf1c824cf" containerName="payload-processing" Nov 04 01:26:35.614118 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.614108 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce052f4b-e92c-4795-a4ba-947cf1c824cf" containerName="payload-processing" Nov 04 01:26:35.614212 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.614207 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea5f3c5-a60c-4781-87de-89edbb87c02a" containerName="payload-pre-processing" Nov 04 01:26:35.614244 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.614218 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce052f4b-e92c-4795-a4ba-947cf1c824cf" containerName="payload-processing" Nov 04 01:26:35.620131 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.620112 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb" Nov 04 01:26:35.625047 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.625021 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m"] Nov 04 01:26:35.629176 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.629158 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m" Nov 04 01:26:35.630625 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.630602 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-af6909\"" Nov 04 01:26:35.634351 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.634332 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-af6909-dockercfg-2c4ck\"" Nov 04 01:26:35.634481 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.634375 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb"] Nov 04 01:26:35.656699 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.656673 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m"] Nov 04 01:26:35.684758 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.684732 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:35.710257 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.710234 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lpn9\" (UniqueName: \"kubernetes.io/projected/ed9eda72-0b3b-4933-bb4d-83d8eee5c20e-kube-api-access-9lpn9\") pod \"payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m\" (UID: \"ed9eda72-0b3b-4933-bb4d-83d8eee5c20e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m" Nov 04 01:26:35.710384 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.710297 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb\" (UID: \"e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb" Nov 04 01:26:35.710507 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.710482 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed9eda72-0b3b-4933-bb4d-83d8eee5c20e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m\" (UID: \"ed9eda72-0b3b-4933-bb4d-83d8eee5c20e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m" Nov 04 01:26:35.710678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.710659 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4k5m\" (UniqueName: \"kubernetes.io/projected/e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed-kube-api-access-g4k5m\") pod \"payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb\" (UID: \"e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb" Nov 04 01:26:35.817773 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.817545 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lpn9\" (UniqueName: \"kubernetes.io/projected/ed9eda72-0b3b-4933-bb4d-83d8eee5c20e-kube-api-access-9lpn9\") pod \"payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m\" (UID: \"ed9eda72-0b3b-4933-bb4d-83d8eee5c20e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m" Nov 04 01:26:35.818387 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.818361 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb\" (UID: \"e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb" Nov 04 01:26:35.818631 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.818611 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed9eda72-0b3b-4933-bb4d-83d8eee5c20e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m\" (UID: \"ed9eda72-0b3b-4933-bb4d-83d8eee5c20e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m" Nov 04 01:26:35.818948 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.818874 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4k5m\" (UniqueName: \"kubernetes.io/projected/e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed-kube-api-access-g4k5m\") pod \"payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb\" (UID: \"e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb" Nov 04 01:26:35.835587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.820615 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb\" (UID: \"e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb" Nov 04 01:26:35.837347 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.837320 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed9eda72-0b3b-4933-bb4d-83d8eee5c20e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m\" (UID: \"ed9eda72-0b3b-4933-bb4d-83d8eee5c20e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m" Nov 04 01:26:35.840514 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.840490 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4k5m\" (UniqueName: \"kubernetes.io/projected/e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed-kube-api-access-g4k5m\") pod \"payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb\" (UID: \"e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb" Nov 04 01:26:35.861653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.855169 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lpn9\" (UniqueName: \"kubernetes.io/projected/ed9eda72-0b3b-4933-bb4d-83d8eee5c20e-kube-api-access-9lpn9\") pod \"payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m\" (UID: \"ed9eda72-0b3b-4933-bb4d-83d8eee5c20e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m" Nov 04 01:26:35.935123 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.935091 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb" Nov 04 01:26:35.970650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:35.958273 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m" Nov 04 01:26:36.309646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:36.309618 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m"] Nov 04 01:26:36.313410 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:36.313384 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb"] Nov 04 01:26:36.313608 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:26:36.313548 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15e5a9d_84ef_4e2b_9b9d_4b5eb1b767ed.slice/crio-732faf5c87d699daa93a0d82ae0366b54fda90efaa245787e65f67832036cb1c WatchSource:0}: Error finding container 732faf5c87d699daa93a0d82ae0366b54fda90efaa245787e65f67832036cb1c: Status 404 returned error can't find the container with id 732faf5c87d699daa93a0d82ae0366b54fda90efaa245787e65f67832036cb1c Nov 04 01:26:36.633165 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:36.628255 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:37.051348 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:37.051311 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m" event={"ID":"ed9eda72-0b3b-4933-bb4d-83d8eee5c20e","Type":"ContainerStarted","Data":"e95d86cbff8b89fd9e939e914ce94b71f371892cfb6d8d7e2cbfbc38949a7883"} Nov 04 01:26:37.051837 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:37.051373 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m" event={"ID":"ed9eda72-0b3b-4933-bb4d-83d8eee5c20e","Type":"ContainerStarted","Data":"8d5900de47eef8558fc223f1940913fab407629d0040a31c11fc16b4a4e5c86a"} Nov 04 01:26:37.051837 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:37.051475 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m" Nov 04 01:26:37.053011 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:37.052985 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb" event={"ID":"e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed","Type":"ContainerStarted","Data":"b3ec36efa7be447a2c582bfb3a0963e666cbfb9a07c949c48ea57447a50fb881"} Nov 04 01:26:37.053145 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:37.053038 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb" Nov 04 01:26:37.053145 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:37.053061 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb" event={"ID":"e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed","Type":"ContainerStarted","Data":"732faf5c87d699daa93a0d82ae0366b54fda90efaa245787e65f67832036cb1c"} Nov 04 01:26:37.075289 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:37.075237 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m" podStartSLOduration=2.075223672 podStartE2EDuration="2.075223672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:37.073026687 +0000 UTC m=+1679.188136807" watchObservedRunningTime="2026-11-04 01:26:37.075223672 +0000 UTC m=+1679.190333828" Nov 04 01:26:37.098364 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:37.098320 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb" podStartSLOduration=2.098303163 podStartE2EDuration="2.098303163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:37.095766567 +0000 UTC m=+1679.210876684" watchObservedRunningTime="2026-11-04 01:26:37.098303163 +0000 UTC m=+1679.213413283" Nov 04 01:26:37.433873 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:37.433737 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:38.116888 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:38.116860 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:38.272258 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:38.272232 4852 scope.go:122] "RemoveContainer" containerID="702e658f6afb66140266c4a22eabea4bb7b0ca67dfbb476045ec386d1cef9929" Nov 04 01:26:38.774319 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:38.774265 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:39.424015 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:39.419818 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:40.046902 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.046859 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:40.189670 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.189629 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:40.560943 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.560906 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:40.631129 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.631099 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx"] Nov 04 01:26:40.637962 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.637934 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" Nov 04 01:26:40.646764 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.646739 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx"] Nov 04 01:26:40.647265 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.647237 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-af6909-serving-cert\"" Nov 04 01:26:40.732804 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.732776 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:40.762932 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.762895 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx\" (UID: \"31ae07d8-c2d3-4732-83ff-b9583ec0fba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" Nov 04 01:26:40.763230 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.763202 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx\" (UID: \"31ae07d8-c2d3-4732-83ff-b9583ec0fba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" Nov 04 01:26:40.763499 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.763474 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxcz7\" (UniqueName: \"kubernetes.io/projected/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-kube-api-access-dxcz7\") pod \"maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx\" (UID: \"31ae07d8-c2d3-4732-83ff-b9583ec0fba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" Nov 04 01:26:40.770894 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.770859 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx\" (UID: \"31ae07d8-c2d3-4732-83ff-b9583ec0fba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" Nov 04 01:26:40.876644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.872945 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxcz7\" (UniqueName: \"kubernetes.io/projected/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-kube-api-access-dxcz7\") pod \"maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx\" (UID: \"31ae07d8-c2d3-4732-83ff-b9583ec0fba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" Nov 04 01:26:40.876644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.873169 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx\" (UID: \"31ae07d8-c2d3-4732-83ff-b9583ec0fba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" Nov 04 01:26:40.876644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.873309 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx\" (UID: \"31ae07d8-c2d3-4732-83ff-b9583ec0fba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" Nov 04 01:26:40.876644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.873345 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx\" (UID: \"31ae07d8-c2d3-4732-83ff-b9583ec0fba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" Nov 04 01:26:40.876644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.874774 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx\" (UID: \"31ae07d8-c2d3-4732-83ff-b9583ec0fba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" Nov 04 01:26:40.879302 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.877945 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:40.882876 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.880641 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx\" (UID: \"31ae07d8-c2d3-4732-83ff-b9583ec0fba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" Nov 04 01:26:40.882876 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.882389 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx\" (UID: \"31ae07d8-c2d3-4732-83ff-b9583ec0fba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" Nov 04 01:26:40.885332 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.885304 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxcz7\" (UniqueName: \"kubernetes.io/projected/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-kube-api-access-dxcz7\") pod \"maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx\" (UID: \"31ae07d8-c2d3-4732-83ff-b9583ec0fba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" Nov 04 01:26:40.960623 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.960550 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj"] Nov 04 01:26:40.961981 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.961803 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" Nov 04 01:26:40.969809 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.969789 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:40.979319 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.979289 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj"] Nov 04 01:26:40.988627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:40.988557 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-016d3d-openshift-default-dockercfg-2ws2r\"" Nov 04 01:26:41.056408 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.056376 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:41.083465 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.077361 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-workload-certs\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.083465 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.077458 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0515d69-ac7d-4d43-a316-58ccc63b757a-istiod-ca-cert\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.083465 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.077537 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-data\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.083465 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.077665 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-token\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.083465 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.077729 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-workload-socket\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.083465 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.077919 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlnpf\" (UniqueName: \"kubernetes.io/projected/b0515d69-ac7d-4d43-a316-58ccc63b757a-kube-api-access-zlnpf\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.083465 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.077948 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-credential-socket\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.083465 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.078012 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-podinfo\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.083465 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.078057 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-envoy\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.213642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.194721 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlnpf\" (UniqueName: \"kubernetes.io/projected/b0515d69-ac7d-4d43-a316-58ccc63b757a-kube-api-access-zlnpf\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.213642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.194812 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-credential-socket\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.213642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.194891 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-podinfo\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.213642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.194943 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-envoy\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.213642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.195049 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-workload-certs\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.213642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.195080 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0515d69-ac7d-4d43-a316-58ccc63b757a-istiod-ca-cert\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.213642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.195161 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-data\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.213642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.195214 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-token\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.213642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.195293 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-workload-socket\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.213642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.195847 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-workload-socket\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.213642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.196409 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-credential-socket\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.276589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.257161 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0515d69-ac7d-4d43-a316-58ccc63b757a-istiod-ca-cert\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.276589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.271945 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-workload-certs\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.339621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.338358 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:41.339621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.338792 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-data\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.351593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.351328 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-envoy\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.429639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.425068 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-podinfo\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.429639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.429206 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlnpf\" (UniqueName: \"kubernetes.io/projected/b0515d69-ac7d-4d43-a316-58ccc63b757a-kube-api-access-zlnpf\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.431188 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.431061 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-token\") pod \"e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.443664 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.443626 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx"] Nov 04 01:26:41.459023 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:26:41.449248 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ae07d8_c2d3_4732_83ff_b9583ec0fba8.slice/crio-710419098e3dc4f587e2fcc50325278df9bd9061bcdddc20a96d6ec6711e630c WatchSource:0}: Error finding container 710419098e3dc4f587e2fcc50325278df9bd9061bcdddc20a96d6ec6711e630c: Status 404 returned error can't find the container with id 710419098e3dc4f587e2fcc50325278df9bd9061bcdddc20a96d6ec6711e630c Nov 04 01:26:41.673597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.647880 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:41.891543 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:41.891201 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:42.078842 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:42.078809 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:42.324772 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:42.324722 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" event={"ID":"31ae07d8-c2d3-4732-83ff-b9583ec0fba8","Type":"ContainerStarted","Data":"710419098e3dc4f587e2fcc50325278df9bd9061bcdddc20a96d6ec6711e630c"} Nov 04 01:26:42.565339 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:42.565314 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj"] Nov 04 01:26:42.566746 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:26:42.566720 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0515d69_ac7d_4d43_a316_58ccc63b757a.slice/crio-3ed4d6d302faba91f9568d1be18a546f25eb932e0ce4e0decae3dd03dd739f50 WatchSource:0}: Error finding container 3ed4d6d302faba91f9568d1be18a546f25eb932e0ce4e0decae3dd03dd739f50: Status 404 returned error can't find the container with id 3ed4d6d302faba91f9568d1be18a546f25eb932e0ce4e0decae3dd03dd739f50 Nov 04 01:26:42.569950 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:42.569918 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:26:42.570028 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:42.569980 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:26:42.570028 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:42.570010 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:26:42.637650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:42.623440 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:43.381585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:43.378402 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:43.424583 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:43.423537 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" event={"ID":"b0515d69-ac7d-4d43-a316-58ccc63b757a","Type":"ContainerStarted","Data":"040ea5bf349d71a7e21d814bbf7efe6e5e85b338a6831db42692d76f4766b5ac"} Nov 04 01:26:43.424583 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:43.423647 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" event={"ID":"b0515d69-ac7d-4d43-a316-58ccc63b757a","Type":"ContainerStarted","Data":"3ed4d6d302faba91f9568d1be18a546f25eb932e0ce4e0decae3dd03dd739f50"} Nov 04 01:26:43.435692 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:43.427908 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" event={"ID":"31ae07d8-c2d3-4732-83ff-b9583ec0fba8","Type":"ContainerStarted","Data":"6c460bccca45abb205209bea483d74d482bc5877b599b771613093bfbac3ba32"} Nov 04 01:26:43.435692 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:43.428939 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" Nov 04 01:26:43.461516 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:43.461456 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" podStartSLOduration=3.461437231 podStartE2EDuration="3.461437231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:43.455080827 +0000 UTC m=+1685.570190946" watchObservedRunningTime="2026-11-04 01:26:43.461437231 +0000 UTC m=+1685.576547352" Nov 04 01:26:43.474381 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:43.474333 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" podStartSLOduration=2.675878399 podStartE2EDuration="3.474317731s" totalImagesPullingTime="798.439332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:43.473419025 +0000 UTC m=+1685.588529144" watchObservedRunningTime="2026-11-04 01:26:43.474317731 +0000 UTC m=+1685.589427848" Nov 04 01:26:43.697715 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:43.688094 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:43.711264 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:43.711233 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:44.125586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:44.124955 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:44.434197 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:44.434107 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:44.434896 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:44.434876 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:26:44.908292 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:44.908255 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:45.562411 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:45.562376 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:45.745557 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:45.745513 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:46.070947 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:46.070905 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:46.641861 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:46.640654 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:47.032324 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.032291 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4"] Nov 04 01:26:47.036393 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.036377 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4" Nov 04 01:26:47.041056 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.041026 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b093e14-419d-48f2-9b07-978b6ad8ae72-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4\" (UID: \"9b093e14-419d-48f2-9b07-978b6ad8ae72\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4" Nov 04 01:26:47.041219 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.041200 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvvr9\" (UniqueName: \"kubernetes.io/projected/9b093e14-419d-48f2-9b07-978b6ad8ae72-kube-api-access-wvvr9\") pod \"payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4\" (UID: \"9b093e14-419d-48f2-9b07-978b6ad8ae72\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4" Nov 04 01:26:47.047117 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.047096 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-016d3d\"" Nov 04 01:26:47.053133 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.053110 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4"] Nov 04 01:26:47.057221 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.057202 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4"] Nov 04 01:26:47.057343 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.057284 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4" Nov 04 01:26:47.075495 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.075472 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-016d3d-dockercfg-9h2l8\"" Nov 04 01:26:47.081213 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.081186 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:47.087447 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.087421 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4"] Nov 04 01:26:47.142609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.142558 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b093e14-419d-48f2-9b07-978b6ad8ae72-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4\" (UID: \"9b093e14-419d-48f2-9b07-978b6ad8ae72\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4" Nov 04 01:26:47.142775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.142630 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fvsg\" (UniqueName: \"kubernetes.io/projected/c85127bb-d834-4223-9736-4416ec320c01-kube-api-access-5fvsg\") pod \"payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4\" (UID: \"c85127bb-d834-4223-9736-4416ec320c01\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4" Nov 04 01:26:47.142775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.142682 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvvr9\" (UniqueName: \"kubernetes.io/projected/9b093e14-419d-48f2-9b07-978b6ad8ae72-kube-api-access-wvvr9\") pod \"payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4\" (UID: \"9b093e14-419d-48f2-9b07-978b6ad8ae72\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4" Nov 04 01:26:47.142869 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.142849 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85127bb-d834-4223-9736-4416ec320c01-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4\" (UID: \"c85127bb-d834-4223-9736-4416ec320c01\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4" Nov 04 01:26:47.143250 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.143232 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b093e14-419d-48f2-9b07-978b6ad8ae72-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4\" (UID: \"9b093e14-419d-48f2-9b07-978b6ad8ae72\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4" Nov 04 01:26:47.164020 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.163983 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvvr9\" (UniqueName: \"kubernetes.io/projected/9b093e14-419d-48f2-9b07-978b6ad8ae72-kube-api-access-wvvr9\") pod \"payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4\" (UID: \"9b093e14-419d-48f2-9b07-978b6ad8ae72\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4" Nov 04 01:26:47.256745 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.255619 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85127bb-d834-4223-9736-4416ec320c01-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4\" (UID: \"c85127bb-d834-4223-9736-4416ec320c01\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4" Nov 04 01:26:47.257020 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.257000 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85127bb-d834-4223-9736-4416ec320c01-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4\" (UID: \"c85127bb-d834-4223-9736-4416ec320c01\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4" Nov 04 01:26:47.257374 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.257354 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fvsg\" (UniqueName: \"kubernetes.io/projected/c85127bb-d834-4223-9736-4416ec320c01-kube-api-access-5fvsg\") pod \"payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4\" (UID: \"c85127bb-d834-4223-9736-4416ec320c01\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4" Nov 04 01:26:47.281306 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.281273 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:47.328422 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.328304 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fvsg\" (UniqueName: \"kubernetes.io/projected/c85127bb-d834-4223-9736-4416ec320c01-kube-api-access-5fvsg\") pod \"payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4\" (UID: \"c85127bb-d834-4223-9736-4416ec320c01\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4" Nov 04 01:26:47.360053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.360023 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4" Nov 04 01:26:47.391626 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.380021 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4" Nov 04 01:26:47.676406 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.649474 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:47.768174 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.768143 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4"] Nov 04 01:26:47.769165 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:26:47.769139 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b093e14_419d_48f2_9b07_978b6ad8ae72.slice/crio-e2ea04fb31f3fdd65f7bf6e54aff45e578eafa5a89a781179c61cf9d0166bec0 WatchSource:0}: Error finding container e2ea04fb31f3fdd65f7bf6e54aff45e578eafa5a89a781179c61cf9d0166bec0: Status 404 returned error can't find the container with id e2ea04fb31f3fdd65f7bf6e54aff45e578eafa5a89a781179c61cf9d0166bec0 Nov 04 01:26:47.886806 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:26:47.886781 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85127bb_d834_4223_9736_4416ec320c01.slice/crio-2b8117a11661c60d91e6f43ea81a13537f729aefc03489dfa470f88edda02111 WatchSource:0}: Error finding container 2b8117a11661c60d91e6f43ea81a13537f729aefc03489dfa470f88edda02111: Status 404 returned error can't find the container with id 2b8117a11661c60d91e6f43ea81a13537f729aefc03489dfa470f88edda02111 Nov 04 01:26:47.887200 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:47.887172 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4"] Nov 04 01:26:48.062490 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:48.062461 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m" Nov 04 01:26:48.062690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:48.062534 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb" Nov 04 01:26:48.268598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:48.267300 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:48.371627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:48.371591 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:26:48.545659 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:48.542813 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4" event={"ID":"c85127bb-d834-4223-9736-4416ec320c01","Type":"ContainerStarted","Data":"bb81cf78b4d5345994bb207725ca4fb1b40355354cd4ab84528f1bba1ea90a4d"} Nov 04 01:26:48.545659 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:48.542883 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4" event={"ID":"c85127bb-d834-4223-9736-4416ec320c01","Type":"ContainerStarted","Data":"2b8117a11661c60d91e6f43ea81a13537f729aefc03489dfa470f88edda02111"} Nov 04 01:26:48.545659 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:48.544007 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4" Nov 04 01:26:48.565473 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:48.565434 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4" event={"ID":"9b093e14-419d-48f2-9b07-978b6ad8ae72","Type":"ContainerStarted","Data":"fae9925231cbd60e4047d81d1548dd4dabc36c736e45c4c38439402e1ab70bfb"} Nov 04 01:26:48.586682 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:48.586641 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4" event={"ID":"9b093e14-419d-48f2-9b07-978b6ad8ae72","Type":"ContainerStarted","Data":"e2ea04fb31f3fdd65f7bf6e54aff45e578eafa5a89a781179c61cf9d0166bec0"} Nov 04 01:26:48.586911 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:48.586895 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4" Nov 04 01:26:48.588899 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:48.588850 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4" podStartSLOduration=1.588833251 podStartE2EDuration="1.588833251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:48.565060354 +0000 UTC m=+1690.680170473" watchObservedRunningTime="2026-11-04 01:26:48.588833251 +0000 UTC m=+1690.703943372" Nov 04 01:26:48.601295 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:48.601239 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4" podStartSLOduration=2.601219767 podStartE2EDuration="2.601219767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:48.60002275 +0000 UTC m=+1690.715132868" watchObservedRunningTime="2026-11-04 01:26:48.601219767 +0000 UTC m=+1690.716329885" Nov 04 01:26:49.179588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:49.176351 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:50.153590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:50.135645 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:50.446066 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:50.445983 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" Nov 04 01:26:51.006540 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:50.998132 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:51.915123 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:51.915091 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:52.728184 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:52.728145 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:53.198068 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:53.198025 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:53.631002 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:53.630788 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:53.811240 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:53.811198 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:54.445898 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:54.445870 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:55.268081 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:55.268043 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:55.967126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:55.967071 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:56.636588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:56.630988 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:57.350587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:57.340762 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:57.969771 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:57.969474 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:59.021484 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:59.021453 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:59.658638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:59.658408 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:59.670105 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:59.670001 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4" Nov 04 01:26:59.795036 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:26:59.795008 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:26:59.952477 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:26:59.952389 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:27:00.375268 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:00.375164 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:00.526386 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:00.526345 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:00.656371 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:00.656284 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4" Nov 04 01:27:00.855150 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:00.855121 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:01.327458 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:01.327432 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:01.544692 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:01.544664 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:02.050131 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:02.050095 4852 generic.go:356] "Generic (PLEG): container finished" podID="9febb536-9d73-41c8-8a6e-5b2aa1446825" containerID="1fe10790ad7324f595f5dc7fc778f0fce0aafce8523825cab82e73850efb7e85" exitCode=0 Nov 04 01:27:02.050374 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:02.050178 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" event={"ID":"9febb536-9d73-41c8-8a6e-5b2aa1446825","Type":"ContainerDied","Data":"1fe10790ad7324f595f5dc7fc778f0fce0aafce8523825cab82e73850efb7e85"} Nov 04 01:27:02.050776 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:02.050751 4852 scope.go:122] "RemoveContainer" containerID="1fe10790ad7324f595f5dc7fc778f0fce0aafce8523825cab82e73850efb7e85" Nov 04 01:27:02.052485 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:02.052453 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:27:02.052661 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:02.052541 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:27:02.052661 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:02.052610 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:27:02.123584 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:02.116262 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:03.011589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:03.004407 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:03.084357 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:03.083001 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" event={"ID":"9febb536-9d73-41c8-8a6e-5b2aa1446825","Type":"ContainerStarted","Data":"43f9bcd7de1208f57d06b688d5017e1b1ee3429c16c6086783527cfbf9f0cea1"} Nov 04 01:27:03.552059 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:03.552031 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:27:04.035586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:04.033789 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:27:04.039074 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:04.037732 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 04 01:27:04.039074 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:04.037792 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" podUID="9febb536-9d73-41c8-8a6e-5b2aa1446825" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 04 01:27:04.088600 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:04.086317 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:04.742876 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:04.742842 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:05.035011 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:05.034969 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 04 01:27:05.035299 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:05.035270 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" podUID="9febb536-9d73-41c8-8a6e-5b2aa1446825" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 04 01:27:05.144704 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:05.144673 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:05.984280 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:05.984245 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:05.990647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:05.990614 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6"] Nov 04 01:27:05.997140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:05.997112 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6" Nov 04 01:27:06.008600 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:06.008552 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-ef6cc9\"" Nov 04 01:27:06.008749 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:06.008676 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-ef6cc9-dockercfg-5hvdx\"" Nov 04 01:27:06.008817 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:06.008741 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6"] Nov 04 01:27:06.033234 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:06.032918 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 04 01:27:06.033234 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:06.032976 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" podUID="9febb536-9d73-41c8-8a6e-5b2aa1446825" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 04 01:27:06.122589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:06.108620 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614a961a-dc3c-41ae-963f-53dfe934eda0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6\" (UID: \"614a961a-dc3c-41ae-963f-53dfe934eda0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6" Nov 04 01:27:06.122589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:06.108786 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhzwg\" (UniqueName: \"kubernetes.io/projected/614a961a-dc3c-41ae-963f-53dfe934eda0-kube-api-access-jhzwg\") pod \"payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6\" (UID: \"614a961a-dc3c-41ae-963f-53dfe934eda0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6" Nov 04 01:27:06.221595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:06.217871 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614a961a-dc3c-41ae-963f-53dfe934eda0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6\" (UID: \"614a961a-dc3c-41ae-963f-53dfe934eda0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6" Nov 04 01:27:06.221595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:06.217995 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhzwg\" (UniqueName: \"kubernetes.io/projected/614a961a-dc3c-41ae-963f-53dfe934eda0-kube-api-access-jhzwg\") pod \"payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6\" (UID: \"614a961a-dc3c-41ae-963f-53dfe934eda0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6" Nov 04 01:27:06.221595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:06.219419 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614a961a-dc3c-41ae-963f-53dfe934eda0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6\" (UID: \"614a961a-dc3c-41ae-963f-53dfe934eda0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6" Nov 04 01:27:06.242376 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:06.242290 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhzwg\" (UniqueName: \"kubernetes.io/projected/614a961a-dc3c-41ae-963f-53dfe934eda0-kube-api-access-jhzwg\") pod \"payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6\" (UID: \"614a961a-dc3c-41ae-963f-53dfe934eda0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6" Nov 04 01:27:06.342596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:06.342223 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:06.347593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:06.343024 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6" Nov 04 01:27:06.790958 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:06.790930 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6"] Nov 04 01:27:06.791522 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:27:06.791496 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614a961a_dc3c_41ae_963f_53dfe934eda0.slice/crio-1bd0a9cd9dea01bc785857f1367f80d7b25082385a1b8e1d29339d7ac9eb50d8 WatchSource:0}: Error finding container 1bd0a9cd9dea01bc785857f1367f80d7b25082385a1b8e1d29339d7ac9eb50d8: Status 404 returned error can't find the container with id 1bd0a9cd9dea01bc785857f1367f80d7b25082385a1b8e1d29339d7ac9eb50d8 Nov 04 01:27:06.928790 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:06.928761 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11014-dbd803"] Nov 04 01:27:06.933305 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:06.933281 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11014-dbd803" Nov 04 01:27:06.944672 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:06.944656 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-rc7dk\"" Nov 04 01:27:06.949556 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:06.949534 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgpds\" (UniqueName: \"kubernetes.io/projected/d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb-kube-api-access-wgpds\") pod \"test-curl-11014-dbd803\" (UID: \"d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb\") " pod="opendatahub/test-curl-11014-dbd803" Nov 04 01:27:07.068649 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:07.059681 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgpds\" (UniqueName: \"kubernetes.io/projected/d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb-kube-api-access-wgpds\") pod \"test-curl-11014-dbd803\" (UID: \"d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb\") " pod="opendatahub/test-curl-11014-dbd803" Nov 04 01:27:07.098775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:07.072025 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 04 01:27:07.099466 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:07.099071 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" podUID="9febb536-9d73-41c8-8a6e-5b2aa1446825" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 04 01:27:07.156211 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:07.156180 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11014-dbd803"] Nov 04 01:27:07.178201 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:07.178150 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:07.182610 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:07.181098 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgpds\" (UniqueName: \"kubernetes.io/projected/d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb-kube-api-access-wgpds\") pod \"test-curl-11014-dbd803\" (UID: \"d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb\") " pod="opendatahub/test-curl-11014-dbd803" Nov 04 01:27:07.245807 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:07.245773 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11014-dbd803" Nov 04 01:27:07.449443 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:07.449408 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6" event={"ID":"614a961a-dc3c-41ae-963f-53dfe934eda0","Type":"ContainerStarted","Data":"d5908a6bcaadf7b071e3c6c45f5f43ef1f8f818254172109b92d6f3a00bdfe06"} Nov 04 01:27:07.449675 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:07.449464 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6" event={"ID":"614a961a-dc3c-41ae-963f-53dfe934eda0","Type":"ContainerStarted","Data":"1bd0a9cd9dea01bc785857f1367f80d7b25082385a1b8e1d29339d7ac9eb50d8"} Nov 04 01:27:07.449675 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:07.449494 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6" Nov 04 01:27:07.483729 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:07.483680 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6" podStartSLOduration=2.483665895 podStartE2EDuration="2.483665895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:07.478909739 +0000 UTC m=+1709.594019858" watchObservedRunningTime="2026-11-04 01:27:07.483665895 +0000 UTC m=+1709.598776012" Nov 04 01:27:07.587647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:07.584443 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11014-dbd803"] Nov 04 01:27:07.588437 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:27:07.588214 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1f6d6e5_6ec0_47ef_8e17_db2841bf0ebb.slice/crio-946ef585ed79f57059298f34fdbce45195ee4d494550131a750b0f03313f54f3 WatchSource:0}: Error finding container 946ef585ed79f57059298f34fdbce45195ee4d494550131a750b0f03313f54f3: Status 404 returned error can't find the container with id 946ef585ed79f57059298f34fdbce45195ee4d494550131a750b0f03313f54f3 Nov 04 01:27:07.839586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:07.835213 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:08.033349 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:08.033316 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 04 01:27:08.033525 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:08.033368 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" podUID="9febb536-9d73-41c8-8a6e-5b2aa1446825" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 04 01:27:08.481591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:08.481094 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11014-dbd803" event={"ID":"d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb","Type":"ContainerStarted","Data":"b99fe70444865bac085e7bb243902ac1882b77bab871407e449759becc4314e1"} Nov 04 01:27:08.481591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:08.481138 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11014-dbd803" event={"ID":"d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb","Type":"ContainerStarted","Data":"946ef585ed79f57059298f34fdbce45195ee4d494550131a750b0f03313f54f3"} Nov 04 01:27:08.502447 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:08.502394 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11014-dbd803" podStartSLOduration=2.502382396 podStartE2EDuration="2.502382396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:08.498899164 +0000 UTC m=+1710.614009283" watchObservedRunningTime="2026-11-04 01:27:08.502382396 +0000 UTC m=+1710.617492513" Nov 04 01:27:08.592588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:08.591528 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:09.033064 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:09.033025 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 04 01:27:09.033267 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:09.033084 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" podUID="9febb536-9d73-41c8-8a6e-5b2aa1446825" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 04 01:27:09.224909 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:09.224875 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11014-dbd803"] Nov 04 01:27:09.237873 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:09.230591 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11014-dbd803"] Nov 04 01:27:09.369070 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:09.368992 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11014-b0f289"] Nov 04 01:27:09.369638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:09.369616 4852 cpu_manager.go:414] "containerMap: removing container" podUID="d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb" containerName="test-curl-11014-dbd803" Nov 04 01:27:09.369638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:09.369633 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb" containerName="test-curl-11014-dbd803" Nov 04 01:27:09.369765 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:09.369756 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb" containerName="test-curl-11014-dbd803" Nov 04 01:27:09.373419 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:09.373404 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11014-b0f289" Nov 04 01:27:09.380000 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:09.379975 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11014-b0f289"] Nov 04 01:27:09.445601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:09.444651 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w99w\" (UniqueName: \"kubernetes.io/projected/4473fe20-ffb6-47b3-a4fc-9ee90a23b905-kube-api-access-9w99w\") pod \"test-curl-11014-b0f289\" (UID: \"4473fe20-ffb6-47b3-a4fc-9ee90a23b905\") " pod="opendatahub/test-curl-11014-b0f289" Nov 04 01:27:09.478482 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:09.478422 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:09.545737 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:09.545705 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w99w\" (UniqueName: \"kubernetes.io/projected/4473fe20-ffb6-47b3-a4fc-9ee90a23b905-kube-api-access-9w99w\") pod \"test-curl-11014-b0f289\" (UID: \"4473fe20-ffb6-47b3-a4fc-9ee90a23b905\") " pod="opendatahub/test-curl-11014-b0f289" Nov 04 01:27:09.555105 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:09.555080 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w99w\" (UniqueName: \"kubernetes.io/projected/4473fe20-ffb6-47b3-a4fc-9ee90a23b905-kube-api-access-9w99w\") pod \"test-curl-11014-b0f289\" (UID: \"4473fe20-ffb6-47b3-a4fc-9ee90a23b905\") " pod="opendatahub/test-curl-11014-b0f289" Nov 04 01:27:09.691051 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:09.690983 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11014-b0f289" Nov 04 01:27:09.966791 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:09.966766 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11014-b0f289"] Nov 04 01:27:09.967337 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:27:09.967313 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4473fe20_ffb6_47b3_a4fc_9ee90a23b905.slice/crio-30a0fe4d150e40b146218b6139ee72260dbbcbb9e518c534be240031747bdad6 WatchSource:0}: Error finding container 30a0fe4d150e40b146218b6139ee72260dbbcbb9e518c534be240031747bdad6: Status 404 returned error can't find the container with id 30a0fe4d150e40b146218b6139ee72260dbbcbb9e518c534be240031747bdad6 Nov 04 01:27:10.037468 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:10.037448 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:27:10.038501 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:10.038483 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:27:10.148424 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:10.148388 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:10.506497 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:10.506463 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11014-b0f289" event={"ID":"4473fe20-ffb6-47b3-a4fc-9ee90a23b905","Type":"ContainerStarted","Data":"146e8b3014498f8470abdf3f6b99d2d60c6d4cd9bdd8f1e5f0fafc03d0eacd3a"} Nov 04 01:27:10.506709 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:10.506516 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11014-b0f289" event={"ID":"4473fe20-ffb6-47b3-a4fc-9ee90a23b905","Type":"ContainerStarted","Data":"30a0fe4d150e40b146218b6139ee72260dbbcbb9e518c534be240031747bdad6"} Nov 04 01:27:10.506938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:10.506912 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11014-dbd803" podUID="d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb" containerName="test-curl-11014-dbd803" containerID="cri-o://b99fe70444865bac085e7bb243902ac1882b77bab871407e449759becc4314e1" gracePeriod=2 Nov 04 01:27:10.523189 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:10.523141 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11014-b0f289" podStartSLOduration=1.523127511 podStartE2EDuration="1.523127511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:10.520156204 +0000 UTC m=+1712.635266323" watchObservedRunningTime="2026-11-04 01:27:10.523127511 +0000 UTC m=+1712.638237628" Nov 04 01:27:10.799498 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:10.789807 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:11.240719 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:11.240616 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11014-b0f289"] Nov 04 01:27:11.244364 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:11.244339 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11014-b0f289"] Nov 04 01:27:11.391377 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:11.391350 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11014-979880"] Nov 04 01:27:11.392037 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:11.392021 4852 cpu_manager.go:414] "containerMap: removing container" podUID="4473fe20-ffb6-47b3-a4fc-9ee90a23b905" containerName="test-curl-11014-b0f289" Nov 04 01:27:11.392080 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:11.392039 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4473fe20-ffb6-47b3-a4fc-9ee90a23b905" containerName="test-curl-11014-b0f289" Nov 04 01:27:11.392139 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:11.392130 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="4473fe20-ffb6-47b3-a4fc-9ee90a23b905" containerName="test-curl-11014-b0f289" Nov 04 01:27:11.395733 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:11.395714 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11014-979880" Nov 04 01:27:11.402815 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:11.402794 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11014-979880"] Nov 04 01:27:11.489587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:11.479894 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95whm\" (UniqueName: \"kubernetes.io/projected/af0bd2af-6e2b-4f12-9f29-5901a144f7f1-kube-api-access-95whm\") pod \"test-curl-11014-979880\" (UID: \"af0bd2af-6e2b-4f12-9f29-5901a144f7f1\") " pod="opendatahub/test-curl-11014-979880" Nov 04 01:27:11.518588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:11.511401 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:11.581171 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:11.581138 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95whm\" (UniqueName: \"kubernetes.io/projected/af0bd2af-6e2b-4f12-9f29-5901a144f7f1-kube-api-access-95whm\") pod \"test-curl-11014-979880\" (UID: \"af0bd2af-6e2b-4f12-9f29-5901a144f7f1\") " pod="opendatahub/test-curl-11014-979880" Nov 04 01:27:11.593006 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:11.592980 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95whm\" (UniqueName: \"kubernetes.io/projected/af0bd2af-6e2b-4f12-9f29-5901a144f7f1-kube-api-access-95whm\") pod \"test-curl-11014-979880\" (UID: \"af0bd2af-6e2b-4f12-9f29-5901a144f7f1\") " pod="opendatahub/test-curl-11014-979880" Nov 04 01:27:11.709915 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:11.709893 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11014-979880" Nov 04 01:27:11.989469 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:27:11.989437 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf0bd2af_6e2b_4f12_9f29_5901a144f7f1.slice/crio-6fd7e308072b121fe4aa5103cb24982d03f0e770cc83beae5af0f63a783a5b0e WatchSource:0}: Error finding container 6fd7e308072b121fe4aa5103cb24982d03f0e770cc83beae5af0f63a783a5b0e: Status 404 returned error can't find the container with id 6fd7e308072b121fe4aa5103cb24982d03f0e770cc83beae5af0f63a783a5b0e Nov 04 01:27:12.000342 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:12.000321 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11014-979880"] Nov 04 01:27:12.153359 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:12.153321 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:12.534850 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:12.534818 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11014-979880" event={"ID":"af0bd2af-6e2b-4f12-9f29-5901a144f7f1","Type":"ContainerStarted","Data":"36d20d5132e9e69fd47549f532d7c1f0fcb791a6f5ec504234dc760f49fbf073"} Nov 04 01:27:12.534969 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:12.534861 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11014-979880" event={"ID":"af0bd2af-6e2b-4f12-9f29-5901a144f7f1","Type":"ContainerStarted","Data":"6fd7e308072b121fe4aa5103cb24982d03f0e770cc83beae5af0f63a783a5b0e"} Nov 04 01:27:12.535036 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:12.535024 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11014-b0f289" podUID="4473fe20-ffb6-47b3-a4fc-9ee90a23b905" containerName="test-curl-11014-b0f289" containerID="cri-o://146e8b3014498f8470abdf3f6b99d2d60c6d4cd9bdd8f1e5f0fafc03d0eacd3a" gracePeriod=2 Nov 04 01:27:12.552882 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:12.552828 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11014-979880" podStartSLOduration=1.552815346 podStartE2EDuration="1.552815346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:12.549682803 +0000 UTC m=+1714.664792924" watchObservedRunningTime="2026-11-04 01:27:12.552815346 +0000 UTC m=+1714.667925464" Nov 04 01:27:12.713278 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:12.713249 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11014-dbd803" Nov 04 01:27:12.808824 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:12.808758 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb-kube-api-access-wgpds\" (UniqueName: \"kubernetes.io/projected/d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb-kube-api-access-wgpds\") pod \"d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb\" (UID: \"d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb\") " Nov 04 01:27:12.810946 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:12.810919 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb-kube-api-access-wgpds" pod "d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb" (UID: "d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb"). InnerVolumeSpecName "kube-api-access-wgpds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:12.823316 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:12.823286 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:12.918601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:12.917148 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgpds\" (UniqueName: \"kubernetes.io/projected/d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb-kube-api-access-wgpds\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:12.995592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:12.994954 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:13.256061 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:13.256032 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11014-979880"] Nov 04 01:27:13.475606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:13.471749 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11014-979880"] Nov 04 01:27:13.526868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:13.526765 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11014-4ab8a6"] Nov 04 01:27:13.528411 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:13.528367 4852 cpu_manager.go:414] "containerMap: removing container" podUID="af0bd2af-6e2b-4f12-9f29-5901a144f7f1" containerName="test-curl-11014-979880" Nov 04 01:27:13.571115 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:13.571076 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af0bd2af-6e2b-4f12-9f29-5901a144f7f1" containerName="test-curl-11014-979880" Nov 04 01:27:13.571953 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:13.571917 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="af0bd2af-6e2b-4f12-9f29-5901a144f7f1" containerName="test-curl-11014-979880" Nov 04 01:27:13.581932 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:13.581908 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11014-4ab8a6"] Nov 04 01:27:13.582229 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:13.582214 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11014-4ab8a6" Nov 04 01:27:13.604599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:13.602173 4852 generic.go:356] "Generic (PLEG): container finished" podID="d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb" containerID="b99fe70444865bac085e7bb243902ac1882b77bab871407e449759becc4314e1" exitCode=137 Nov 04 01:27:13.604599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:13.603422 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11014-dbd803" Nov 04 01:27:13.605353 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:13.604900 4852 scope.go:122] "RemoveContainer" containerID="b99fe70444865bac085e7bb243902ac1882b77bab871407e449759becc4314e1" Nov 04 01:27:13.635869 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:27:13.635820 4852 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11014-dbd803\" is forbidden: User \"system:node:ip-10-0-140-241.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-241.us-east-2.compute.internal' and this object" podUID="d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb" pod="opendatahub/test-curl-11014-dbd803" Nov 04 01:27:13.640986 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:13.640953 4852 scope.go:122] "RemoveContainer" containerID="b99fe70444865bac085e7bb243902ac1882b77bab871407e449759becc4314e1" Nov 04 01:27:13.641443 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:27:13.641394 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b99fe70444865bac085e7bb243902ac1882b77bab871407e449759becc4314e1\": container with ID starting with b99fe70444865bac085e7bb243902ac1882b77bab871407e449759becc4314e1 not found: ID does not exist" containerID="b99fe70444865bac085e7bb243902ac1882b77bab871407e449759becc4314e1" Nov 04 01:27:13.641443 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:13.641421 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b99fe70444865bac085e7bb243902ac1882b77bab871407e449759becc4314e1"} err="failed to get container status \"b99fe70444865bac085e7bb243902ac1882b77bab871407e449759becc4314e1\": rpc error: code = NotFound desc = could not find container \"b99fe70444865bac085e7bb243902ac1882b77bab871407e449759becc4314e1\": container with ID starting with b99fe70444865bac085e7bb243902ac1882b77bab871407e449759becc4314e1 not found: ID does not exist" Nov 04 01:27:13.741605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:13.730784 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hdnp\" (UniqueName: \"kubernetes.io/projected/c858f6da-c146-4412-8ba7-4e82b7839ea8-kube-api-access-9hdnp\") pod \"test-curl-11014-4ab8a6\" (UID: \"c858f6da-c146-4412-8ba7-4e82b7839ea8\") " pod="opendatahub/test-curl-11014-4ab8a6" Nov 04 01:27:13.802794 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:13.800180 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:13.840589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:13.836698 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdnp\" (UniqueName: \"kubernetes.io/projected/c858f6da-c146-4412-8ba7-4e82b7839ea8-kube-api-access-9hdnp\") pod \"test-curl-11014-4ab8a6\" (UID: \"c858f6da-c146-4412-8ba7-4e82b7839ea8\") " pod="opendatahub/test-curl-11014-4ab8a6" Nov 04 01:27:13.886585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:13.886225 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hdnp\" (UniqueName: \"kubernetes.io/projected/c858f6da-c146-4412-8ba7-4e82b7839ea8-kube-api-access-9hdnp\") pod \"test-curl-11014-4ab8a6\" (UID: \"c858f6da-c146-4412-8ba7-4e82b7839ea8\") " pod="opendatahub/test-curl-11014-4ab8a6" Nov 04 01:27:13.943587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:13.942962 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11014-4ab8a6" Nov 04 01:27:14.014604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:14.014556 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb" path="/var/lib/kubelet/pods/d1f6d6e5-6ec0-47ef-8e17-db2841bf0ebb/volumes" Nov 04 01:27:14.267804 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:14.267772 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11014-4ab8a6"] Nov 04 01:27:14.268530 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:27:14.268507 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc858f6da_c146_4412_8ba7_4e82b7839ea8.slice/crio-c6a7f22c30ceebbc09ff99d8ac7ec31de01cb2b6a9b4bdd2b3ac04ced735050d WatchSource:0}: Error finding container c6a7f22c30ceebbc09ff99d8ac7ec31de01cb2b6a9b4bdd2b3ac04ced735050d: Status 404 returned error can't find the container with id c6a7f22c30ceebbc09ff99d8ac7ec31de01cb2b6a9b4bdd2b3ac04ced735050d Nov 04 01:27:14.523673 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:14.521861 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:14.630250 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:14.630216 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11014-4ab8a6" event={"ID":"c858f6da-c146-4412-8ba7-4e82b7839ea8","Type":"ContainerStarted","Data":"ab5ca987437415d50188cc6a60ba4ded61cc44f8296d9d36afd2bc85e011a05d"} Nov 04 01:27:14.630427 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:14.630263 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11014-4ab8a6" event={"ID":"c858f6da-c146-4412-8ba7-4e82b7839ea8","Type":"ContainerStarted","Data":"c6a7f22c30ceebbc09ff99d8ac7ec31de01cb2b6a9b4bdd2b3ac04ced735050d"} Nov 04 01:27:14.630427 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:14.630367 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11014-979880" podUID="af0bd2af-6e2b-4f12-9f29-5901a144f7f1" containerName="test-curl-11014-979880" containerID="cri-o://36d20d5132e9e69fd47549f532d7c1f0fcb791a6f5ec504234dc760f49fbf073" gracePeriod=2 Nov 04 01:27:14.645917 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:14.645863 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11014-4ab8a6" podStartSLOduration=1.645846409 podStartE2EDuration="1.645846409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:14.644311344 +0000 UTC m=+1716.759421459" watchObservedRunningTime="2026-11-04 01:27:14.645846409 +0000 UTC m=+1716.760956527" Nov 04 01:27:14.756090 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:14.756066 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11014-b0f289" Nov 04 01:27:14.762758 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:14.762736 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4473fe20-ffb6-47b3-a4fc-9ee90a23b905-kube-api-access-9w99w\" (UniqueName: \"kubernetes.io/projected/4473fe20-ffb6-47b3-a4fc-9ee90a23b905-kube-api-access-9w99w\") pod \"4473fe20-ffb6-47b3-a4fc-9ee90a23b905\" (UID: \"4473fe20-ffb6-47b3-a4fc-9ee90a23b905\") " Nov 04 01:27:14.765263 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:14.765240 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4473fe20-ffb6-47b3-a4fc-9ee90a23b905-kube-api-access-9w99w" pod "4473fe20-ffb6-47b3-a4fc-9ee90a23b905" (UID: "4473fe20-ffb6-47b3-a4fc-9ee90a23b905"). InnerVolumeSpecName "kube-api-access-9w99w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:14.865100 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:14.865014 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w99w\" (UniqueName: \"kubernetes.io/projected/4473fe20-ffb6-47b3-a4fc-9ee90a23b905-kube-api-access-9w99w\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:15.191647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:15.180713 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:15.301252 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:15.301220 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11014-4ab8a6"] Nov 04 01:27:15.304990 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:15.304961 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11014-4ab8a6"] Nov 04 01:27:15.638239 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:15.638209 4852 generic.go:356] "Generic (PLEG): container finished" podID="4473fe20-ffb6-47b3-a4fc-9ee90a23b905" containerID="146e8b3014498f8470abdf3f6b99d2d60c6d4cd9bdd8f1e5f0fafc03d0eacd3a" exitCode=137 Nov 04 01:27:15.638437 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:15.638252 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11014-b0f289" Nov 04 01:27:15.638437 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:15.638315 4852 scope.go:122] "RemoveContainer" containerID="146e8b3014498f8470abdf3f6b99d2d60c6d4cd9bdd8f1e5f0fafc03d0eacd3a" Nov 04 01:27:15.647800 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:15.647787 4852 scope.go:122] "RemoveContainer" containerID="146e8b3014498f8470abdf3f6b99d2d60c6d4cd9bdd8f1e5f0fafc03d0eacd3a" Nov 04 01:27:15.648067 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:27:15.648050 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"146e8b3014498f8470abdf3f6b99d2d60c6d4cd9bdd8f1e5f0fafc03d0eacd3a\": container with ID starting with 146e8b3014498f8470abdf3f6b99d2d60c6d4cd9bdd8f1e5f0fafc03d0eacd3a not found: ID does not exist" containerID="146e8b3014498f8470abdf3f6b99d2d60c6d4cd9bdd8f1e5f0fafc03d0eacd3a" Nov 04 01:27:15.648121 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:15.648074 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"146e8b3014498f8470abdf3f6b99d2d60c6d4cd9bdd8f1e5f0fafc03d0eacd3a"} err="failed to get container status \"146e8b3014498f8470abdf3f6b99d2d60c6d4cd9bdd8f1e5f0fafc03d0eacd3a\": rpc error: code = NotFound desc = could not find container \"146e8b3014498f8470abdf3f6b99d2d60c6d4cd9bdd8f1e5f0fafc03d0eacd3a\": container with ID starting with 146e8b3014498f8470abdf3f6b99d2d60c6d4cd9bdd8f1e5f0fafc03d0eacd3a not found: ID does not exist" Nov 04 01:27:15.649025 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:27:15.649007 4852 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11014-b0f289\" is forbidden: User \"system:node:ip-10-0-140-241.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-241.us-east-2.compute.internal' and this object" podUID="4473fe20-ffb6-47b3-a4fc-9ee90a23b905" pod="opendatahub/test-curl-11014-b0f289" Nov 04 01:27:15.864664 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:15.861438 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:15.996185 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:15.996103 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4473fe20-ffb6-47b3-a4fc-9ee90a23b905" path="/var/lib/kubelet/pods/4473fe20-ffb6-47b3-a4fc-9ee90a23b905/volumes" Nov 04 01:27:16.642488 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:16.642447 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:16.711131 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:16.711082 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11014-4ab8a6" podUID="c858f6da-c146-4412-8ba7-4e82b7839ea8" containerName="test-curl-11014-4ab8a6" containerID="cri-o://ab5ca987437415d50188cc6a60ba4ded61cc44f8296d9d36afd2bc85e011a05d" gracePeriod=2 Nov 04 01:27:16.884055 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:16.884032 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11014-979880" Nov 04 01:27:16.985490 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:16.985415 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af0bd2af-6e2b-4f12-9f29-5901a144f7f1-kube-api-access-95whm\" (UniqueName: \"kubernetes.io/projected/af0bd2af-6e2b-4f12-9f29-5901a144f7f1-kube-api-access-95whm\") pod \"af0bd2af-6e2b-4f12-9f29-5901a144f7f1\" (UID: \"af0bd2af-6e2b-4f12-9f29-5901a144f7f1\") " Nov 04 01:27:16.987606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:16.987581 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0bd2af-6e2b-4f12-9f29-5901a144f7f1-kube-api-access-95whm" pod "af0bd2af-6e2b-4f12-9f29-5901a144f7f1" (UID: "af0bd2af-6e2b-4f12-9f29-5901a144f7f1"). InnerVolumeSpecName "kube-api-access-95whm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:17.087282 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.087254 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95whm\" (UniqueName: \"kubernetes.io/projected/af0bd2af-6e2b-4f12-9f29-5901a144f7f1-kube-api-access-95whm\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:17.132898 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.132869 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:17.339598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.339225 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:17.517652 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.517621 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j"] Nov 04 01:27:17.518391 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.518372 4852 cpu_manager.go:414] "containerMap: removing container" podUID="c858f6da-c146-4412-8ba7-4e82b7839ea8" containerName="test-curl-11014-4ab8a6" Nov 04 01:27:17.518391 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.518391 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c858f6da-c146-4412-8ba7-4e82b7839ea8" containerName="test-curl-11014-4ab8a6" Nov 04 01:27:17.518657 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.518490 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="c858f6da-c146-4412-8ba7-4e82b7839ea8" containerName="test-curl-11014-4ab8a6" Nov 04 01:27:17.521823 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.521802 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.528712 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.528402 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3888c434-openshift-default-dockercfg-lbhwx\"" Nov 04 01:27:17.532680 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.531924 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j"] Nov 04 01:27:17.669643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.669130 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-envoy\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.669643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.669180 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvp9q\" (UniqueName: \"kubernetes.io/projected/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-kube-api-access-rvp9q\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.669643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.669207 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-podinfo\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.669643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.669237 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-credential-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.669643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.669322 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-data\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.669643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.669346 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istiod-ca-cert\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.669643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.669520 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-token\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.670413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.669664 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-workload-certs\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.670413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.669726 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-workload-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.683587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.678527 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:17.792592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.787893 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-data\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.792592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.787940 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istiod-ca-cert\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.792592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.788205 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-token\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.792592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.788330 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-workload-certs\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.792592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.788389 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-workload-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.792592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.788471 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-envoy\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.792592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.788499 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvp9q\" (UniqueName: \"kubernetes.io/projected/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-kube-api-access-rvp9q\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.792592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.788522 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-podinfo\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.792592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.788554 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-credential-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.792592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.789016 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-credential-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.792592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.789294 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-data\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.792592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.790021 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istiod-ca-cert\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.801592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.799328 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-workload-certs\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.801592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.799775 4852 generic.go:356] "Generic (PLEG): container finished" podID="af0bd2af-6e2b-4f12-9f29-5901a144f7f1" containerID="36d20d5132e9e69fd47549f532d7c1f0fcb791a6f5ec504234dc760f49fbf073" exitCode=137 Nov 04 01:27:17.801592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.800049 4852 scope.go:122] "RemoveContainer" containerID="36d20d5132e9e69fd47549f532d7c1f0fcb791a6f5ec504234dc760f49fbf073" Nov 04 01:27:17.801592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.800210 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11014-979880" Nov 04 01:27:17.844588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.841918 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-envoy\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.844588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.842036 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-workload-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.871619 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.871080 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-podinfo\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.889593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.885603 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvp9q\" (UniqueName: \"kubernetes.io/projected/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-kube-api-access-rvp9q\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:17.889593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:17.886078 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-token\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:18.032475 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:18.032303 4852 scope.go:122] "RemoveContainer" containerID="36d20d5132e9e69fd47549f532d7c1f0fcb791a6f5ec504234dc760f49fbf073" Nov 04 01:27:18.037714 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:27:18.034713 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d20d5132e9e69fd47549f532d7c1f0fcb791a6f5ec504234dc760f49fbf073\": container with ID starting with 36d20d5132e9e69fd47549f532d7c1f0fcb791a6f5ec504234dc760f49fbf073 not found: ID does not exist" containerID="36d20d5132e9e69fd47549f532d7c1f0fcb791a6f5ec504234dc760f49fbf073" Nov 04 01:27:18.037714 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:18.034754 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d20d5132e9e69fd47549f532d7c1f0fcb791a6f5ec504234dc760f49fbf073"} err="failed to get container status \"36d20d5132e9e69fd47549f532d7c1f0fcb791a6f5ec504234dc760f49fbf073\": rpc error: code = NotFound desc = could not find container \"36d20d5132e9e69fd47549f532d7c1f0fcb791a6f5ec504234dc760f49fbf073\": container with ID starting with 36d20d5132e9e69fd47549f532d7c1f0fcb791a6f5ec504234dc760f49fbf073 not found: ID does not exist" Nov 04 01:27:18.041665 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:27:18.041488 4852 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11014-979880\" is forbidden: User \"system:node:ip-10-0-140-241.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-241.us-east-2.compute.internal' and this object" podUID="af0bd2af-6e2b-4f12-9f29-5901a144f7f1" pod="opendatahub/test-curl-11014-979880" Nov 04 01:27:18.053607 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:18.053316 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af0bd2af-6e2b-4f12-9f29-5901a144f7f1" path="/var/lib/kubelet/pods/af0bd2af-6e2b-4f12-9f29-5901a144f7f1/volumes" Nov 04 01:27:18.142855 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:18.142816 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:18.291949 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:18.287689 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:18.467335 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:18.467304 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:18.469492 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:18.469450 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:27:18.469649 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:18.469612 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:27:18.469745 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:18.469726 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:27:18.479806 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:18.479783 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j"] Nov 04 01:27:18.484994 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:18.484958 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6" Nov 04 01:27:18.826970 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:18.826902 4852 generic.go:356] "Generic (PLEG): container finished" podID="c858f6da-c146-4412-8ba7-4e82b7839ea8" containerID="ab5ca987437415d50188cc6a60ba4ded61cc44f8296d9d36afd2bc85e011a05d" exitCode=137 Nov 04 01:27:18.830609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:18.830580 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" event={"ID":"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a","Type":"ContainerStarted","Data":"7373d1b8c9784e142253cb4d2cb5ab006a4159586628f7ea493d25eae8604db7"} Nov 04 01:27:18.830789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:18.830772 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" event={"ID":"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a","Type":"ContainerStarted","Data":"a8ad8d36c60bd7b3dc086249553323905dcc28e0e9b5774ca1eadc5897ad17c7"} Nov 04 01:27:18.853041 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:18.852925 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" podStartSLOduration=1.852909852 podStartE2EDuration="1.852909852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:18.852091705 +0000 UTC m=+1720.967201823" watchObservedRunningTime="2026-11-04 01:27:18.852909852 +0000 UTC m=+1720.968019974" Nov 04 01:27:18.929536 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:18.929512 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11014-4ab8a6" Nov 04 01:27:19.000429 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:19.000390 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c858f6da-c146-4412-8ba7-4e82b7839ea8-kube-api-access-9hdnp\" (UniqueName: \"kubernetes.io/projected/c858f6da-c146-4412-8ba7-4e82b7839ea8-kube-api-access-9hdnp\") pod \"c858f6da-c146-4412-8ba7-4e82b7839ea8\" (UID: \"c858f6da-c146-4412-8ba7-4e82b7839ea8\") " Nov 04 01:27:19.018884 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:19.018816 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c858f6da-c146-4412-8ba7-4e82b7839ea8-kube-api-access-9hdnp" pod "c858f6da-c146-4412-8ba7-4e82b7839ea8" (UID: "c858f6da-c146-4412-8ba7-4e82b7839ea8"). InnerVolumeSpecName "kube-api-access-9hdnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:19.103614 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:19.103410 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hdnp\" (UniqueName: \"kubernetes.io/projected/c858f6da-c146-4412-8ba7-4e82b7839ea8-kube-api-access-9hdnp\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:19.154592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:19.150936 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:19.183589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:19.182879 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:19.257765 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:19.257392 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:19.839642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:19.839614 4852 scope.go:122] "RemoveContainer" containerID="ab5ca987437415d50188cc6a60ba4ded61cc44f8296d9d36afd2bc85e011a05d" Nov 04 01:27:19.839642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:19.839630 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11014-4ab8a6" Nov 04 01:27:19.840168 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:19.840089 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:19.841152 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:19.841129 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:27:19.862913 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:27:19.862885 4852 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11014-4ab8a6\" is forbidden: User \"system:node:ip-10-0-140-241.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-241.us-east-2.compute.internal' and this object" podUID="c858f6da-c146-4412-8ba7-4e82b7839ea8" pod="opendatahub/test-curl-11014-4ab8a6" Nov 04 01:27:19.930425 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:19.930400 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:20.046674 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:20.046631 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c858f6da-c146-4412-8ba7-4e82b7839ea8" path="/var/lib/kubelet/pods/c858f6da-c146-4412-8ba7-4e82b7839ea8/volumes" Nov 04 01:27:20.729589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:20.722669 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:21.227625 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:21.227587 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:21.384782 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:21.384751 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:21.643608 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:21.639949 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:22.342605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:22.340040 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:22.512970 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:22.512939 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:23.483225 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:23.483187 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:24.263331 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:24.263279 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:24.823550 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:24.823517 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:25.362606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:25.362549 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:26.294263 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:26.294231 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:27.164234 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:27.164118 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:27.236737 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:27.236688 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:27.330588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:27.326670 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:27.339591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:27.332026 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:27.553716 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:27.553655 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:27.577704 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:27.577657 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:27.614066 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:27.614024 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:27:27.631597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:27.630605 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:27.750588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:27.749726 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:27.880649 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:27.876643 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Nov 04 01:27:27.959689 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:27.959642 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:27.964664 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:27.964630 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:28.238315 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:28.238092 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:28.238801 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:28.238715 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:28.240345 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:28.239159 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:27:28.514042 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:28.513997 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:27:28.521739 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:28.520635 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:27:28.531592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:28.531151 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:27:28.558612 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:28.557449 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:28.558612 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:28.557636 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:28.558612 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:28.558528 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:27:28.581982 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:28.580150 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:28.581982 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:28.580665 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:27:28.619621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:28.617297 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:28.619621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:28.617882 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:27:28.657696 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:28.646908 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:28.657696 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:28.647627 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:28.657696 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:28.648085 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:28.657696 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:28.648097 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:27:28.723418 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:28.716534 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:27:28.753196 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:28.750206 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 04 01:27:28.791654 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:28.776017 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:27:29.041127 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:29.037964 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:27:29.041127 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:29.039536 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:27:29.164647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:29.146168 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:29.244918 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:29.244790 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:27:29.595663 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:29.525657 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:27:29.595663 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:29.566464 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:27:29.595663 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:29.567266 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:27:29.595663 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:29.590944 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:29.665591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:29.657658 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:29.665591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:29.658169 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:29.665591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:29.658650 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:27:29.770428 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:29.752712 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:29.770428 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:29.766695 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:29.927651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:29.912131 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": context deadline exceeded" Nov 04 01:27:30.020596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:30.015943 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:30.091589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:30.076896 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:27:30.252352 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:30.252250 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:30.312153 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:30.311399 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:30.535941 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:30.535888 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:30.547902 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:30.547817 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:30.575056 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:30.575002 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 04 01:27:30.576951 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:30.576486 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded" Nov 04 01:27:30.598790 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:30.598018 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 04 01:27:30.673642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:30.659981 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:30.673642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:30.661300 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 04 01:27:30.702080 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:27:30.676337 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:27:30.702080 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:30.676677 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:30.754219 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:30.754173 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:30.789047 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:30.788950 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:27:30.789971 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:30.789946 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:27:30.958493 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:30.958455 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:27:31.079643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:31.077735 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:31.079828 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:31.079787 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:31.306798 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:31.306718 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:31.307381 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:31.307344 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:27:31.326588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:31.320796 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:27:31.326588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:31.322550 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:27:31.382131 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:31.356456 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:27:31.382131 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:31.368250 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:27:31.421629 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:31.415911 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:27:31.449851 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:31.447797 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:27:31.625639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:31.622982 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:27:31.627960 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:31.627933 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:27:31.738245 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:31.738153 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:31.980638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:31.979875 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:32.658623 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:32.658555 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:33.234588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:33.232381 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:33.463020 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:33.452057 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:34.245664 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:34.245551 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:34.953038 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:34.952699 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:27:34.953038 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:34.952764 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" podUID="d31799de-2b0d-4f2e-839e-058668f304c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:35.042819 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.042782 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:35.217679 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.217591 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:35.554362 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.554322 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw"] Nov 04 01:27:35.562815 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.562787 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.570493 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.569861 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-2fff32-openshift-default-dockercfg-x2msp\"" Nov 04 01:27:35.572031 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.571988 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw"] Nov 04 01:27:35.646045 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.646015 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:35.671549 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.671515 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-token\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.671722 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.671585 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-data\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.671836 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.671808 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-credential-socket\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.671905 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.671888 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-workload-certs\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.672129 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.672110 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zvk9\" (UniqueName: \"kubernetes.io/projected/1efe83d4-74d6-431e-91fd-314ce0f5d116-kube-api-access-9zvk9\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.672282 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.672263 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-podinfo\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.672432 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.672413 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-envoy\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.672481 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.672470 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1efe83d4-74d6-431e-91fd-314ce0f5d116-istiod-ca-cert\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.672723 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.672701 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-workload-socket\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.796972 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.781729 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-workload-socket\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.796972 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.781792 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-token\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.796972 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.781826 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-data\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.796972 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.781935 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-credential-socket\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.796972 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.781980 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-workload-certs\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.796972 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.782109 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zvk9\" (UniqueName: \"kubernetes.io/projected/1efe83d4-74d6-431e-91fd-314ce0f5d116-kube-api-access-9zvk9\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.796972 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.782219 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-podinfo\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.796972 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.782320 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-envoy\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.796972 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.782359 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1efe83d4-74d6-431e-91fd-314ce0f5d116-istiod-ca-cert\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.796972 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.783287 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1efe83d4-74d6-431e-91fd-314ce0f5d116-istiod-ca-cert\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.796972 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.783586 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-workload-socket\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.796972 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.784121 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-data\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.796972 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.784347 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-credential-socket\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.796972 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.784590 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-workload-certs\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.850537 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.829449 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-token\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.850537 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.840819 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-envoy\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.874083 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.862713 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-podinfo\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.897587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.895618 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:35.897587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.897359 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zvk9\" (UniqueName: \"kubernetes.io/projected/1efe83d4-74d6-431e-91fd-314ce0f5d116-kube-api-access-9zvk9\") pod \"e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:35.907881 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:35.904930 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:36.152649 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:36.149364 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:36.628336 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:36.628304 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw"] Nov 04 01:27:36.629330 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:27:36.629304 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1efe83d4_74d6_431e_91fd_314ce0f5d116.slice/crio-3900b6741ad20b15c0389824907b063dc6205ff734e79fc863965dbee899b3ac WatchSource:0}: Error finding container 3900b6741ad20b15c0389824907b063dc6205ff734e79fc863965dbee899b3ac: Status 404 returned error can't find the container with id 3900b6741ad20b15c0389824907b063dc6205ff734e79fc863965dbee899b3ac Nov 04 01:27:36.632877 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:36.632838 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:27:36.632992 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:36.632924 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:27:36.632992 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:36.632965 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:27:36.913339 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:36.913200 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:36.940014 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:36.937467 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" event={"ID":"1efe83d4-74d6-431e-91fd-314ce0f5d116","Type":"ContainerStarted","Data":"392fea8c004cca56e05fd1f56f459bc5f07ec94d8d4b6daf9bb2fb2fb7596c91"} Nov 04 01:27:36.940014 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:36.937533 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" event={"ID":"1efe83d4-74d6-431e-91fd-314ce0f5d116","Type":"ContainerStarted","Data":"3900b6741ad20b15c0389824907b063dc6205ff734e79fc863965dbee899b3ac"} Nov 04 01:27:37.059593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:37.056276 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" podStartSLOduration=2.056255392 podStartE2EDuration="2.056255392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:36.962610073 +0000 UTC m=+1739.077720191" watchObservedRunningTime="2026-11-04 01:27:37.056255392 +0000 UTC m=+1739.171365508" Nov 04 01:27:37.797411 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:37.797367 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:37.906549 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:37.906423 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:37.915076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:37.913822 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:37.946482 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:37.946448 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:37.947642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:37.947619 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:27:38.808284 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:38.808250 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:38.954028 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:38.953928 4852 generic.go:356] "Generic (PLEG): container finished" podID="b0515d69-ac7d-4d43-a316-58ccc63b757a" containerID="040ea5bf349d71a7e21d814bbf7efe6e5e85b338a6831db42692d76f4766b5ac" exitCode=0 Nov 04 01:27:38.954028 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:38.953999 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" event={"ID":"b0515d69-ac7d-4d43-a316-58ccc63b757a","Type":"ContainerDied","Data":"040ea5bf349d71a7e21d814bbf7efe6e5e85b338a6831db42692d76f4766b5ac"} Nov 04 01:27:38.954597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:38.954533 4852 scope.go:122] "RemoveContainer" containerID="040ea5bf349d71a7e21d814bbf7efe6e5e85b338a6831db42692d76f4766b5ac" Nov 04 01:27:38.955990 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:38.955970 4852 generic.go:356] "Generic (PLEG): container finished" podID="d31799de-2b0d-4f2e-839e-058668f304c6" containerID="09ce9b824a363adf76899b88511527a5e5b7bed4d59032c212dafa11238668f7" exitCode=0 Nov 04 01:27:38.956098 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:38.956014 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" event={"ID":"d31799de-2b0d-4f2e-839e-058668f304c6","Type":"ContainerDied","Data":"09ce9b824a363adf76899b88511527a5e5b7bed4d59032c212dafa11238668f7"} Nov 04 01:27:38.956470 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:38.956439 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:27:38.956511 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:38.956501 4852 scope.go:122] "RemoveContainer" containerID="09ce9b824a363adf76899b88511527a5e5b7bed4d59032c212dafa11238668f7" Nov 04 01:27:38.956545 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:38.956528 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:27:38.956620 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:38.956601 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:27:38.957941 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:38.957913 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:27:38.958032 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:38.957970 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:27:38.958032 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:38.958004 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:27:39.429161 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:39.429125 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:39.589197 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:39.589163 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:39.806401 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:39.796006 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:40.024308 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:40.024272 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" event={"ID":"d31799de-2b0d-4f2e-839e-058668f304c6","Type":"ContainerStarted","Data":"c617da0f9451141112c806b4771869ea40c51f980e5bb4b1f7b207d843c5e4f2"} Nov 04 01:27:40.024748 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:40.024336 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" event={"ID":"b0515d69-ac7d-4d43-a316-58ccc63b757a","Type":"ContainerStarted","Data":"784c828947250e74ae3aecf94b2e833fcd4ed99ecebc4a02f1cb562ecc8e840c"} Nov 04 01:27:40.555599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:40.551072 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:40.713436 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:40.695383 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:27:40.713436 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:40.710453 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 04 01:27:40.713436 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:40.710518 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" podUID="b0515d69-ac7d-4d43-a316-58ccc63b757a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 04 01:27:40.728550 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:40.728516 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:40.745218 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:40.745188 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9"] Nov 04 01:27:40.748883 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:40.748862 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9" Nov 04 01:27:40.944656 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:40.923153 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-2fff32-dockercfg-gl2n5\"" Nov 04 01:27:40.944656 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:40.923307 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-2fff32\"" Nov 04 01:27:41.015586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:40.980328 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:27:41.015586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:40.985265 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:27:41.015586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:40.985317 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" podUID="d31799de-2b0d-4f2e-839e-058668f304c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 04 01:27:41.176273 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:41.176232 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldcxv\" (UniqueName: \"kubernetes.io/projected/f3ab7ef8-0104-4e1f-8327-41964669c3b3-kube-api-access-ldcxv\") pod \"payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9\" (UID: \"f3ab7ef8-0104-4e1f-8327-41964669c3b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9" Nov 04 01:27:41.227463 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:41.227375 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3ab7ef8-0104-4e1f-8327-41964669c3b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9\" (UID: \"f3ab7ef8-0104-4e1f-8327-41964669c3b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9" Nov 04 01:27:41.319728 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:41.319691 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9"] Nov 04 01:27:41.329168 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:41.329128 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldcxv\" (UniqueName: \"kubernetes.io/projected/f3ab7ef8-0104-4e1f-8327-41964669c3b3-kube-api-access-ldcxv\") pod \"payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9\" (UID: \"f3ab7ef8-0104-4e1f-8327-41964669c3b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9" Nov 04 01:27:41.329335 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:41.329269 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3ab7ef8-0104-4e1f-8327-41964669c3b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9\" (UID: \"f3ab7ef8-0104-4e1f-8327-41964669c3b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9" Nov 04 01:27:41.330113 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:41.330094 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3ab7ef8-0104-4e1f-8327-41964669c3b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9\" (UID: \"f3ab7ef8-0104-4e1f-8327-41964669c3b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9" Nov 04 01:27:41.337886 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:41.337865 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldcxv\" (UniqueName: \"kubernetes.io/projected/f3ab7ef8-0104-4e1f-8327-41964669c3b3-kube-api-access-ldcxv\") pod \"payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9\" (UID: \"f3ab7ef8-0104-4e1f-8327-41964669c3b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9" Nov 04 01:27:41.523587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:41.517661 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:41.593652 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:41.592692 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9" Nov 04 01:27:41.702593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:41.701010 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 04 01:27:41.702593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:41.701092 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" podUID="b0515d69-ac7d-4d43-a316-58ccc63b757a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 04 01:27:41.979589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:41.965927 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:27:41.979589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:41.965985 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" podUID="d31799de-2b0d-4f2e-839e-058668f304c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 04 01:27:42.117849 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:42.117167 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9"] Nov 04 01:27:42.227596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:42.225156 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:42.387652 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:42.385670 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9" event={"ID":"f3ab7ef8-0104-4e1f-8327-41964669c3b3","Type":"ContainerStarted","Data":"8119424db22250b2054357f70df9c305418f7f03f163f98a512e30053c727380"} Nov 04 01:27:42.687647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:42.686880 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 04 01:27:42.687647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:42.686938 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" podUID="b0515d69-ac7d-4d43-a316-58ccc63b757a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 04 01:27:42.895005 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:42.894969 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:42.970379 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:42.970297 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:27:42.970665 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:42.970633 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" podUID="d31799de-2b0d-4f2e-839e-058668f304c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 04 01:27:43.417977 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:43.417941 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9" event={"ID":"f3ab7ef8-0104-4e1f-8327-41964669c3b3","Type":"ContainerStarted","Data":"79820189a70b7167614eb3153c36a1eaf94bb1834ef31df1e257b37bce3106e4"} Nov 04 01:27:43.418505 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:43.418234 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9" Nov 04 01:27:43.437908 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:43.437193 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9" podStartSLOduration=3.437179753 podStartE2EDuration="3.437179753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:43.434467939 +0000 UTC m=+1745.549578057" watchObservedRunningTime="2026-11-04 01:27:43.437179753 +0000 UTC m=+1745.552289870" Nov 04 01:27:43.697640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:43.694010 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 04 01:27:43.697640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:43.694081 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" podUID="b0515d69-ac7d-4d43-a316-58ccc63b757a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 04 01:27:43.735678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:43.735638 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:43.974649 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:43.971688 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:27:43.974649 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:43.971744 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" podUID="d31799de-2b0d-4f2e-839e-058668f304c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 04 01:27:44.325828 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:44.325789 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:44.434200 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:44.434174 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:27:44.689005 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:44.688910 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 04 01:27:44.689259 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:44.689226 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" podUID="b0515d69-ac7d-4d43-a316-58ccc63b757a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 04 01:27:44.728073 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:44.728042 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:44.922709 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:44.920898 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:27:44.965635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:44.963927 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:27:44.965635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:44.963984 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" podUID="d31799de-2b0d-4f2e-839e-058668f304c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 04 01:27:45.584604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:45.580999 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:45.690477 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:45.690443 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 04 01:27:45.690793 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:45.690764 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" podUID="b0515d69-ac7d-4d43-a316-58ccc63b757a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 04 01:27:45.966937 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:45.966851 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:27:45.967192 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:45.967164 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" podUID="d31799de-2b0d-4f2e-839e-058668f304c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 04 01:27:46.289353 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:46.289305 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:46.686777 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:46.686665 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 04 01:27:46.686777 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:46.686730 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" podUID="b0515d69-ac7d-4d43-a316-58ccc63b757a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 04 01:27:46.788416 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:46.788386 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:46.944393 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:46.944288 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:46.969647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:46.969621 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:27:46.970158 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:46.970140 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:27:46.970912 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:46.970891 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:27:47.327085 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:47.327051 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:47.691468 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:47.691390 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:27:47.692395 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:47.692377 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:27:47.817284 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:47.817242 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz"] Nov 04 01:27:47.840236 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:47.840204 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:47.841050 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:47.841025 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz"] Nov 04 01:27:47.856242 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:47.856163 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3888c434\"/\"kube-root-ca.crt\"" Nov 04 01:27:47.856381 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:47.856362 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3888c434\"/\"openshift-service-ca.crt\"" Nov 04 01:27:47.857851 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:47.857828 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3888c434\"/\"default-dockercfg-t29n9\"" Nov 04 01:27:47.858074 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:47.858055 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3888c434\"/\"e2e-lifecycle-model-3888c434-kserve-self-signed-certs\"" Nov 04 01:27:47.927309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:47.927282 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:48.007696 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:48.007667 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-home\") pod \"e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:48.011768 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:48.011701 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-model-cache\") pod \"e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:48.011940 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:48.011838 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-tls-certs\") pod \"e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:48.011940 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:48.011885 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8nfr\" (UniqueName: \"kubernetes.io/projected/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-kube-api-access-s8nfr\") pod \"e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:48.012059 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:48.011956 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-dshm\") pod \"e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:48.012059 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:48.012013 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-tmp-dir\") pod \"e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:48.119391 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:48.119357 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-home\") pod \"e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:48.120158 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:48.120135 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-model-cache\") pod \"e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:48.120412 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:48.120392 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-tls-certs\") pod \"e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:48.120581 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:48.120544 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8nfr\" (UniqueName: \"kubernetes.io/projected/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-kube-api-access-s8nfr\") pod \"e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:48.120763 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:48.120746 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-dshm\") pod \"e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:48.120910 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:48.120893 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-tmp-dir\") pod \"e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:48.121474 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:48.121454 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-tmp-dir\") pod \"e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:48.121623 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:48.120035 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-home\") pod \"e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:48.121966 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:48.121947 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-model-cache\") pod \"e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:48.139480 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:48.139447 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-dshm\") pod \"e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:48.145520 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:48.145417 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8nfr\" (UniqueName: \"kubernetes.io/projected/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-kube-api-access-s8nfr\") pod \"e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:48.146667 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:48.146599 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-tls-certs\") pod \"e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:48.168270 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:48.168245 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:48.241587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:48.236091 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:48.614394 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:48.614366 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz"] Nov 04 01:27:48.615205 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:27:48.615177 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1aea2bc_8d86_4c48_8ce7_e0182f02328b.slice/crio-3c40d6f2dbe1c70c19a7a727da90ba100b1b662edb0eb556f5cdb918b64362fc WatchSource:0}: Error finding container 3c40d6f2dbe1c70c19a7a727da90ba100b1b662edb0eb556f5cdb918b64362fc: Status 404 returned error can't find the container with id 3c40d6f2dbe1c70c19a7a727da90ba100b1b662edb0eb556f5cdb918b64362fc Nov 04 01:27:48.849502 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:48.849425 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:49.515584 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:49.509405 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" event={"ID":"b1aea2bc-8d86-4c48-8ce7-e0182f02328b","Type":"ContainerStarted","Data":"d6d46115bafd6a2bae1a5dd623d90c4af5a38fedf1bd43c6c4a1e487176147ed"} Nov 04 01:27:49.515584 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:49.509465 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" event={"ID":"b1aea2bc-8d86-4c48-8ce7-e0182f02328b","Type":"ContainerStarted","Data":"3c40d6f2dbe1c70c19a7a727da90ba100b1b662edb0eb556f5cdb918b64362fc"} Nov 04 01:27:49.535483 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:49.535419 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" podStartSLOduration=2.535406391 podStartE2EDuration="2.535406391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:49.531906441 +0000 UTC m=+1751.647016558" watchObservedRunningTime="2026-11-04 01:27:49.535406391 +0000 UTC m=+1751.650516508" Nov 04 01:27:49.592813 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:49.592758 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:49.749634 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:49.749552 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:49.923213 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:49.923106 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:50.176629 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:50.176528 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:50.631610 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:50.629971 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:50.865201 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:50.865155 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:51.398141 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:51.398084 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:51.561634 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:51.561245 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:52.142673 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:52.142640 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:52.937655 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:52.937615 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:53.654589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:53.651353 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:54.428660 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:54.428636 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9" Nov 04 01:27:54.541307 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:54.541280 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:55.262247 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:55.262201 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:55.981853 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:55.981809 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:56.739724 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:56.739683 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:57.085642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:57.084423 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:57.574601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:57.572257 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:58.168579 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:58.168530 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:58.181913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:58.181892 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:58.258586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:58.249902 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:58.714235 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:58.714209 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:27:59.051349 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:59.051246 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:59.753538 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:27:59.753501 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:27:59.870449 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:27:59.870411 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:28:00.655588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.655197 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:00.701045 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.700992 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x"] Nov 04 01:28:00.742279 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.742251 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x"] Nov 04 01:28:00.742713 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.742616 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.806362 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.806330 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.806859 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.806395 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.806859 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.806537 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.806859 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.806618 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.806859 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.806718 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hsjs\" (UniqueName: \"kubernetes.io/projected/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-kube-api-access-6hsjs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.806859 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.806781 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.807168 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.806864 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.807168 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.807048 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.807168 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.807100 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.833487 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.833465 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-ef6cc9-openshift-default-dockercfg-m7xbv\"" Nov 04 01:28:00.908465 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.908397 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.908465 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.908443 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.908693 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.908516 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.908693 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.908595 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.908693 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.908647 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.908693 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.908676 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.908963 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.908745 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hsjs\" (UniqueName: \"kubernetes.io/projected/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-kube-api-access-6hsjs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.908963 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.908799 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.908963 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.908843 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.908963 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.908853 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.908963 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.908872 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.908963 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.908946 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.909192 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.909019 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.909447 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.909427 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.911095 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.911064 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.911390 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.911373 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.917353 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.917324 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:00.917419 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:00.917369 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hsjs\" (UniqueName: \"kubernetes.io/projected/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-kube-api-access-6hsjs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:01.063127 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:01.063098 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:01.456492 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:01.455819 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x"] Nov 04 01:28:01.478834 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:28:01.475990 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f385f89_e8a1_42b1_a6e1_839aaf1c3530.slice/crio-4ca3a8ba2f14659d48ff2af473731c7af65db835d4a1c4a25dfe0f4f7edfbd16 WatchSource:0}: Error finding container 4ca3a8ba2f14659d48ff2af473731c7af65db835d4a1c4a25dfe0f4f7edfbd16: Status 404 returned error can't find the container with id 4ca3a8ba2f14659d48ff2af473731c7af65db835d4a1c4a25dfe0f4f7edfbd16 Nov 04 01:28:01.478834 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:01.478230 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:01.478834 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:01.478299 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:01.478834 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:01.478339 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:01.577592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:01.577489 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:01.750241 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:01.750146 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" event={"ID":"5f385f89-e8a1-42b1-a6e1-839aaf1c3530","Type":"ContainerStarted","Data":"c24a651f23c86541edda8f10c95a3b3f90937f808bfaa308d9ed393b96dd271d"} Nov 04 01:28:01.750241 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:01.750206 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" event={"ID":"5f385f89-e8a1-42b1-a6e1-839aaf1c3530","Type":"ContainerStarted","Data":"4ca3a8ba2f14659d48ff2af473731c7af65db835d4a1c4a25dfe0f4f7edfbd16"} Nov 04 01:28:01.770468 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:01.770140 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podStartSLOduration=1.770124298 podStartE2EDuration="1.770124298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:01.769753881 +0000 UTC m=+1763.884864001" watchObservedRunningTime="2026-11-04 01:28:01.770124298 +0000 UTC m=+1763.885234417" Nov 04 01:28:02.063916 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:02.063882 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:02.065250 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:02.065227 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:02.065351 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:02.065280 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:02.483916 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:02.483792 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:03.064083 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:03.064056 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:03.064427 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:03.064102 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:03.248638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:03.242546 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:04.032421 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:04.032385 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:04.066016 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:04.065442 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:04.066016 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:04.065529 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:04.170250 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:04.170222 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:04.842149 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:04.842121 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:05.033006 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.032968 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:05.063523 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.063492 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:05.063696 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.063535 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:05.196662 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.196603 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:05.436238 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.436155 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4"] Nov 04 01:28:05.441251 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.441225 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.444371 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.444327 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4"] Nov 04 01:28:05.454674 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.454638 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-98f6f6-openshift-default-dockercfg-c99wm\"" Nov 04 01:28:05.463918 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.463891 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:05.580760 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.563876 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-workload-certs\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.580760 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.563979 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-podinfo\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.580760 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.564114 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9ln9\" (UniqueName: \"kubernetes.io/projected/c72aca29-e3f2-4a8f-abea-8841337cc569-kube-api-access-r9ln9\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.580760 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.564172 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c72aca29-e3f2-4a8f-abea-8841337cc569-istiod-ca-cert\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.580760 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.564218 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-data\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.580760 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.564303 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-credential-socket\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.580760 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.564365 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-token\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.580760 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.564392 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-workload-socket\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.580760 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.564429 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-envoy\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.629250 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.629174 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:05.673987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.671728 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-credential-socket\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.673987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.671807 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-token\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.673987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.671839 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-workload-socket\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.673987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.671875 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-envoy\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.673987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.671943 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-workload-certs\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.673987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.671977 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-podinfo\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.673987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.672133 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9ln9\" (UniqueName: \"kubernetes.io/projected/c72aca29-e3f2-4a8f-abea-8841337cc569-kube-api-access-r9ln9\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.673987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.672212 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c72aca29-e3f2-4a8f-abea-8841337cc569-istiod-ca-cert\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.673987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.672261 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-data\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.673987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.672823 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-data\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.673987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.673077 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-credential-socket\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.673987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.673673 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-workload-socket\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.676852 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.675953 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-workload-certs\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.677300 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.677279 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c72aca29-e3f2-4a8f-abea-8841337cc569-istiod-ca-cert\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.696200 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.696015 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-token\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.696200 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.696158 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9ln9\" (UniqueName: \"kubernetes.io/projected/c72aca29-e3f2-4a8f-abea-8841337cc569-kube-api-access-r9ln9\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.696780 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.696736 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-envoy\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.703640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.703057 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-podinfo\") pod \"e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:05.771707 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:05.759841 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:06.063867 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:06.063832 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:06.064047 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:06.063884 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:06.100503 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:28:06.100476 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72aca29_e3f2_4a8f_abea_8841337cc569.slice/crio-ce0d5542f158588e32a825e4b37813b1525a1f43894e4b4c9ab36fe460ac7de4 WatchSource:0}: Error finding container ce0d5542f158588e32a825e4b37813b1525a1f43894e4b4c9ab36fe460ac7de4: Status 404 returned error can't find the container with id ce0d5542f158588e32a825e4b37813b1525a1f43894e4b4c9ab36fe460ac7de4 Nov 04 01:28:06.101285 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:06.101260 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4"] Nov 04 01:28:06.103196 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:06.103168 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:06.103264 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:06.103228 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:06.103264 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:06.103256 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:06.367109 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:06.367021 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:06.835526 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:06.835485 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" event={"ID":"c72aca29-e3f2-4a8f-abea-8841337cc569","Type":"ContainerStarted","Data":"25d8e1ca785c573a8219c4cfff8b56925602c99c8b31a7953c1d3f17e234ac6e"} Nov 04 01:28:06.835739 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:06.835546 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" event={"ID":"c72aca29-e3f2-4a8f-abea-8841337cc569","Type":"ContainerStarted","Data":"ce0d5542f158588e32a825e4b37813b1525a1f43894e4b4c9ab36fe460ac7de4"} Nov 04 01:28:06.862841 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:06.862794 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podStartSLOduration=1.862781075 podStartE2EDuration="1.862781075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:06.86107542 +0000 UTC m=+1768.976185539" watchObservedRunningTime="2026-11-04 01:28:06.862781075 +0000 UTC m=+1768.977891193" Nov 04 01:28:07.041588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:07.031052 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:07.068600 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:07.066969 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:07.068600 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:07.067026 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:07.760495 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:07.760460 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:07.762074 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:07.762047 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:28:07.762234 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:07.762101 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:28:08.064388 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:08.064317 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:08.064388 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:08.064369 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:08.761392 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:08.761362 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:28:08.761790 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:08.761411 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:28:09.064001 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:09.063931 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:09.064001 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:09.063978 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:09.761385 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:09.761353 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:28:09.761677 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:09.761408 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:28:10.064286 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:10.064208 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:10.064286 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:10.064264 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:10.761084 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:10.761048 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:28:10.761262 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:10.761100 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:28:11.064090 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:11.064015 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:11.064090 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:11.064060 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:11.761292 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:11.761261 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:28:11.761476 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:11.761312 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:28:12.064394 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:12.064309 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:12.064394 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:12.064361 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:12.760429 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:12.760395 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:28:12.760623 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:12.760450 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:28:13.064441 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:13.064360 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:13.064441 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:13.064420 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:13.761104 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:13.761062 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:28:13.761276 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:13.761113 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:28:14.064156 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:14.064082 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:14.064156 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:14.064129 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:14.761055 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:14.761026 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:28:14.761453 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:14.761078 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:28:15.064366 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.064288 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:15.064366 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.064337 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:15.302803 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.302759 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:15.739079 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.739047 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv"] Nov 04 01:28:15.744665 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.744646 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:15.757449 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.757421 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv"] Nov 04 01:28:15.760958 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.760932 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:28:15.761166 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.761133 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:28:15.771256 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.771232 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn"] Nov 04 01:28:15.776645 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.776625 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:15.790972 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.790951 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn"] Nov 04 01:28:15.897014 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.896987 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqrnr\" (UniqueName: \"kubernetes.io/projected/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-kube-api-access-hqrnr\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:15.897147 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.897022 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:15.897147 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.897048 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:15.897251 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.897231 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:15.897298 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.897275 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:15.897402 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.897386 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:15.897457 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.897428 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:15.897496 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.897458 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:15.897545 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.897525 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:15.897606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.897576 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:15.897606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.897595 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:15.897676 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.897661 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:15.897710 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.897679 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:15.897710 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.897698 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:15.897776 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.897740 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:15.897810 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.897781 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5mcw\" (UniqueName: \"kubernetes.io/projected/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-kube-api-access-j5mcw\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:15.897810 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.897804 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:15.897912 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.897855 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:15.999153 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.999076 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5mcw\" (UniqueName: \"kubernetes.io/projected/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-kube-api-access-j5mcw\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:15.999153 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.999120 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:15.999356 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.999243 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:15.999356 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.999284 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqrnr\" (UniqueName: \"kubernetes.io/projected/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-kube-api-access-hqrnr\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:15.999356 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.999305 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:15.999356 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.999340 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:15.999619 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.999446 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:15.999619 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.999493 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:15.999619 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.999500 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:15.999619 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.999610 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:15.999848 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.999674 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:15.999848 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.999697 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:15.999848 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.999721 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:15.999848 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.999776 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:15.999848 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.999808 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:16.000098 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.999862 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:16.000098 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.999867 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:16.000098 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.999923 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:16.000098 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.999948 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:16.000098 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:15.999976 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:16.000098 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.000015 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:16.000098 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.000034 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:16.000098 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.000014 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:16.000718 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.000225 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:16.000718 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.000248 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:16.000718 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.000478 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:16.000718 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.000713 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:16.000943 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.000797 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:16.002600 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.002483 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:16.002713 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.002695 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:16.002759 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.002709 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:16.002759 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.002698 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:16.010104 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.010078 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqrnr\" (UniqueName: \"kubernetes.io/projected/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-kube-api-access-hqrnr\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:16.017989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.017966 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:16.020885 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.020858 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:16.021034 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.021009 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5mcw\" (UniqueName: \"kubernetes.io/projected/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-kube-api-access-j5mcw\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:16.063718 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.063695 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:16.063985 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.063951 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:16.064073 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.063997 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:16.085077 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.085055 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:16.095579 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.095537 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:16.379080 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.379056 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv"] Nov 04 01:28:16.379840 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:28:16.379816 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f0b9b3_5fd3_4857_967f_a18a6fb481d7.slice/crio-6c8a03ef643c4496a6ffd7bf1f7551f7883d81facc25d8f41dbb0645840211a9 WatchSource:0}: Error finding container 6c8a03ef643c4496a6ffd7bf1f7551f7883d81facc25d8f41dbb0645840211a9: Status 404 returned error can't find the container with id 6c8a03ef643c4496a6ffd7bf1f7551f7883d81facc25d8f41dbb0645840211a9 Nov 04 01:28:16.382804 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.382757 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:16.382938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.382819 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:16.382938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.382844 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:16.388026 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:28:16.388002 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84bf0b4_1021_4c15_8ce0_e7b77b11f850.slice/crio-258f8635d42971be0f31ac465be636768e3d92a2885b16aa5171da748651821e WatchSource:0}: Error finding container 258f8635d42971be0f31ac465be636768e3d92a2885b16aa5171da748651821e: Status 404 returned error can't find the container with id 258f8635d42971be0f31ac465be636768e3d92a2885b16aa5171da748651821e Nov 04 01:28:16.389944 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.389926 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn"] Nov 04 01:28:16.393537 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.393508 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:16.393634 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.393580 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:16.393634 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.393609 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:16.760822 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.760787 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:28:16.761001 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:16.760852 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:28:17.075080 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:17.074080 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:17.075080 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:17.074146 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:17.104588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:17.101085 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" event={"ID":"71f0b9b3-5fd3-4857-967f-a18a6fb481d7","Type":"ContainerStarted","Data":"0d64beb5917027fc312d6eaa92b4ed17095674258edb6ecbdad6b50843d2dba0"} Nov 04 01:28:17.104588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:17.101170 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" event={"ID":"71f0b9b3-5fd3-4857-967f-a18a6fb481d7","Type":"ContainerStarted","Data":"6c8a03ef643c4496a6ffd7bf1f7551f7883d81facc25d8f41dbb0645840211a9"} Nov 04 01:28:17.108636 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:17.106083 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:17.113212 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:17.111250 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:28:17.113212 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:17.111314 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 04 01:28:17.113212 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:17.111869 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" event={"ID":"f84bf0b4-1021-4c15-8ce0-e7b77b11f850","Type":"ContainerStarted","Data":"4d3e9f4714237102b221f4ce40a934c4eebe7fa18ca6ef148f5fe30b0aed4a28"} Nov 04 01:28:17.113212 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:17.111919 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" event={"ID":"f84bf0b4-1021-4c15-8ce0-e7b77b11f850","Type":"ContainerStarted","Data":"258f8635d42971be0f31ac465be636768e3d92a2885b16aa5171da748651821e"} Nov 04 01:28:17.137999 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:17.137948 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" podStartSLOduration=2.13793328 podStartE2EDuration="2.13793328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:17.1351867 +0000 UTC m=+1779.250296819" watchObservedRunningTime="2026-11-04 01:28:17.13793328 +0000 UTC m=+1779.253043397" Nov 04 01:28:17.171002 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:17.170949 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" podStartSLOduration=2.170935683 podStartE2EDuration="2.170935683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:17.168788267 +0000 UTC m=+1779.283898386" watchObservedRunningTime="2026-11-04 01:28:17.170935683 +0000 UTC m=+1779.286045858" Nov 04 01:28:17.760819 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:17.760782 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:28:17.761003 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:17.760848 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:28:17.867366 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:17.867330 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:18.063403 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:18.063324 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:18.063403 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:18.063378 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:18.085427 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:18.085391 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:28:18.085944 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:18.085456 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 04 01:28:18.095754 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:18.095728 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:18.097433 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:18.097401 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:28:18.097584 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:18.097440 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 04 01:28:18.760697 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:18.760661 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:28:18.760874 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:18.760711 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:28:18.799334 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:18.799309 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:18.992628 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:18.984905 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:19.064214 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:19.064138 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:19.064214 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:19.064185 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:19.086072 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:19.086043 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:28:19.086409 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:19.086088 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 04 01:28:19.096649 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:19.096623 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:28:19.096744 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:19.096667 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 04 01:28:19.404386 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:19.404310 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:19.558415 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:19.558388 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:19.728053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:19.727871 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:19.789645 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:19.760629 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:28:19.789645 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:19.760716 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:28:20.063833 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:20.063804 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:20.064052 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:20.063855 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:20.085623 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:20.085598 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:28:20.085729 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:20.085643 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 04 01:28:20.096727 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:20.096709 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:28:20.097038 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:20.096762 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 04 01:28:20.404652 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:20.404547 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:20.761157 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:20.761096 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:20.761345 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:20.761324 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:28:20.761387 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:20.761368 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:28:21.060882 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:21.060795 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:21.082091 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:21.082043 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:21.082382 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:21.082351 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:21.087074 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:21.087050 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:28:21.087294 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:21.087257 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 04 01:28:21.106422 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:21.106391 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:28:21.107070 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:21.107035 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 04 01:28:21.507321 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:21.507276 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:21.761160 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:21.761080 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:28:21.761160 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:21.761130 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:28:21.867505 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:21.867463 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:22.064202 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:22.064121 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:22.064202 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:22.064186 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:22.086143 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:22.086109 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:28:22.086312 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:22.086209 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 04 01:28:22.096796 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:22.096772 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:28:22.096922 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:22.096812 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 04 01:28:22.396890 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:22.396809 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:22.762644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:22.760877 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:28:22.762644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:22.760950 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:28:22.792370 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:22.792334 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:23.063642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:23.063508 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:23.063642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:23.063574 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:23.086257 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:23.086216 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:28:23.086428 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:23.086272 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 04 01:28:23.096745 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:23.096719 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:28:23.096898 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:23.096765 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 04 01:28:23.761734 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:23.761693 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:28:23.762329 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:23.762298 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:28:23.882927 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:23.882886 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:24.063977 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:24.063894 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:24.063977 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:24.063943 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:24.085643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:24.085613 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:28:24.085774 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:24.085669 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 04 01:28:24.096394 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:24.096362 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:28:24.096537 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:24.096396 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 04 01:28:24.687093 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:24.687054 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:24.760385 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:24.760354 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:28:24.760538 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:24.760400 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:28:25.063347 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:25.063319 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:25.063736 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:25.063363 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:25.086042 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:25.086014 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:28:25.086144 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:25.086062 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 04 01:28:25.095945 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:25.095917 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:28:25.096048 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:25.095961 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 04 01:28:25.371105 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:25.368436 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:25.760864 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:25.760842 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:28:25.761024 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:25.760890 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:28:26.063416 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:26.063345 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:26.063416 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:26.063390 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:26.085455 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:26.085423 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:28:26.085609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:26.085467 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 04 01:28:26.099331 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:26.099304 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:28:26.099445 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:26.099360 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 04 01:28:26.249528 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:26.249497 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:26.760545 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:26.760511 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:28:26.760756 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:26.760587 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:28:27.058635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:27.058536 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:27.063521 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:27.063500 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:27.063863 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:27.063545 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:27.085615 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:27.085590 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:28:27.085801 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:27.085642 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 04 01:28:27.095875 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:27.095848 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:28:27.095982 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:27.095892 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 04 01:28:27.765589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:27.761948 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:28:27.765589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:27.762023 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:28:27.774940 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:27.774902 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:28.064243 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:28.064114 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:28.064243 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:28.064160 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:28.085744 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:28.085715 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:28:28.085925 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:28.085763 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 04 01:28:28.096112 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:28.096087 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:28:28.096222 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:28.096142 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 04 01:28:28.421645 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:28.419463 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:28.766256 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:28.766227 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:28.767490 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:28.767467 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:28:29.063653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:29.063544 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:29.063653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:29.063625 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:29.086106 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:29.086078 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:28:29.086464 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:29.086121 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 04 01:28:29.096086 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:29.096061 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:28:29.096196 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:29.096106 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 04 01:28:29.227915 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:29.227875 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:29.867100 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:28:29.867063 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:28:29.965589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:29.951240 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:30.063721 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:30.063685 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:30.063887 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:30.063739 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:30.085368 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:30.085339 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:28:30.085532 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:30.085380 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 04 01:28:30.096841 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:30.096817 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:28:30.097304 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:30.096857 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 04 01:28:30.645148 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:30.645110 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:31.063728 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:31.063693 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:31.063917 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:31.063743 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:31.063917 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:31.063784 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:31.064221 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:31.064202 4852 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="istio-proxy" containerStatusID={"Type":"cri-o","ID":"c24a651f23c86541edda8f10c95a3b3f90937f808bfaa308d9ed393b96dd271d"} pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" containerMessage="Container istio-proxy failed startup probe, will be restarted" Nov 04 01:28:31.064262 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:31.064247 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" containerID="cri-o://c24a651f23c86541edda8f10c95a3b3f90937f808bfaa308d9ed393b96dd271d" gracePeriod=30 Nov 04 01:28:31.085972 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:31.085944 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:31.086142 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:31.086122 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:28:31.086220 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:31.086166 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 04 01:28:31.096372 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:31.096353 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:31.096544 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:31.096525 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:28:31.096641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:31.096596 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 04 01:28:31.335335 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:31.335209 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:32.085430 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:32.085397 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:28:32.085618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:32.085451 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 04 01:28:32.096340 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:32.096309 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:28:32.096457 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:32.096347 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 04 01:28:32.201353 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:32.197703 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:33.086060 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:33.086014 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:28:33.086435 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:33.086076 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 04 01:28:33.096402 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:33.096378 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:28:33.096539 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:33.096424 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 04 01:28:33.163404 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:33.163299 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:33.847171 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:33.847133 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:34.096091 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:34.096057 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:28:34.096671 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:34.096495 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 04 01:28:34.096671 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:34.096104 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:28:34.097321 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:34.097247 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 04 01:28:34.477784 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:34.477693 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:34.741214 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:34.741126 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:35.085440 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:35.085410 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:28:35.085623 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:35.085459 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 04 01:28:35.102428 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:35.102405 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:35.103516 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:35.103496 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:28:35.606542 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:35.606509 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:36.090153 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:36.090131 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:36.091194 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:36.091179 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:28:36.188504 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:36.188461 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:36.188930 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:36.188547 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:36.188930 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:36.188616 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:36.323752 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:36.323628 4852 generic.go:356] "Generic (PLEG): container finished" podID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerID="c24a651f23c86541edda8f10c95a3b3f90937f808bfaa308d9ed393b96dd271d" exitCode=0 Nov 04 01:28:36.326281 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:36.325171 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" event={"ID":"5f385f89-e8a1-42b1-a6e1-839aaf1c3530","Type":"ContainerDied","Data":"c24a651f23c86541edda8f10c95a3b3f90937f808bfaa308d9ed393b96dd271d"} Nov 04 01:28:36.326281 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:36.325227 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" event={"ID":"5f385f89-e8a1-42b1-a6e1-839aaf1c3530","Type":"ContainerStarted","Data":"b35f53e833e0d72738cf6e6f08c7c522edbd7f720f752b37886e33581bcbfce7"} Nov 04 01:28:36.903811 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:36.903662 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:37.064208 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:37.064176 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:37.064383 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:37.064353 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:28:37.064460 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:37.064403 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 04 01:28:37.506044 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:37.506013 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:37.707831 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:37.707800 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:37.887942 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:37.887826 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:38.104313 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:38.104004 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:38.127908 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:38.106707 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:38.233179 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:38.233106 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 01:28:38.233488 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:38.233471 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 01:28:38.340876 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:38.340843 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:38.341763 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:38.341741 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:28:38.957679 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:38.957643 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc"] Nov 04 01:28:38.965365 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:38.965339 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc" Nov 04 01:28:38.973130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:38.973102 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc"] Nov 04 01:28:38.976356 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:38.976333 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:38.977092 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:38.977066 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-98f6f6\"" Nov 04 01:28:39.050612 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:39.050545 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vthzw\" (UniqueName: \"kubernetes.io/projected/7af43160-27ea-4e98-8e54-46e9a0adc36c-kube-api-access-vthzw\") pod \"payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc\" (UID: \"7af43160-27ea-4e98-8e54-46e9a0adc36c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc" Nov 04 01:28:39.050897 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:39.050872 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7af43160-27ea-4e98-8e54-46e9a0adc36c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc\" (UID: \"7af43160-27ea-4e98-8e54-46e9a0adc36c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc" Nov 04 01:28:39.152616 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:39.152585 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vthzw\" (UniqueName: \"kubernetes.io/projected/7af43160-27ea-4e98-8e54-46e9a0adc36c-kube-api-access-vthzw\") pod \"payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc\" (UID: \"7af43160-27ea-4e98-8e54-46e9a0adc36c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc" Nov 04 01:28:39.152777 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:39.152731 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7af43160-27ea-4e98-8e54-46e9a0adc36c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc\" (UID: \"7af43160-27ea-4e98-8e54-46e9a0adc36c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc" Nov 04 01:28:39.153518 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:39.153491 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7af43160-27ea-4e98-8e54-46e9a0adc36c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc\" (UID: \"7af43160-27ea-4e98-8e54-46e9a0adc36c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc" Nov 04 01:28:39.165983 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:39.165959 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vthzw\" (UniqueName: \"kubernetes.io/projected/7af43160-27ea-4e98-8e54-46e9a0adc36c-kube-api-access-vthzw\") pod \"payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc\" (UID: \"7af43160-27ea-4e98-8e54-46e9a0adc36c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc" Nov 04 01:28:39.288599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:39.284015 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc" Nov 04 01:28:39.832415 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:39.832386 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc"] Nov 04 01:28:39.833190 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:28:39.833164 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af43160_27ea_4e98_8e54_46e9a0adc36c.slice/crio-66a9061a64de21f088789c5ba01b4320ac82cb004c58847b81a8f78a725c84e8 WatchSource:0}: Error finding container 66a9061a64de21f088789c5ba01b4320ac82cb004c58847b81a8f78a725c84e8: Status 404 returned error can't find the container with id 66a9061a64de21f088789c5ba01b4320ac82cb004c58847b81a8f78a725c84e8 Nov 04 01:28:40.054712 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:40.054686 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq"] Nov 04 01:28:40.058682 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:40.058661 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b61ce18-429mq"] Nov 04 01:28:40.437049 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:40.437010 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc" event={"ID":"7af43160-27ea-4e98-8e54-46e9a0adc36c","Type":"ContainerStarted","Data":"4d30b0a9f7bda93093a0561d1eca450ef39d245c4a36484bb9679208b39b3a3f"} Nov 04 01:28:40.437238 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:40.437072 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc" event={"ID":"7af43160-27ea-4e98-8e54-46e9a0adc36c","Type":"ContainerStarted","Data":"66a9061a64de21f088789c5ba01b4320ac82cb004c58847b81a8f78a725c84e8"} Nov 04 01:28:40.437238 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:40.437103 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc" Nov 04 01:28:40.515762 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:40.515698 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc" podStartSLOduration=2.515676467 podStartE2EDuration="2.515676467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:40.482874431 +0000 UTC m=+1802.597984552" watchObservedRunningTime="2026-11-04 01:28:40.515676467 +0000 UTC m=+1802.630786586" Nov 04 01:28:41.360225 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:41.360197 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:42.046586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:42.042437 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="863dc71f-0017-4800-81a0-053404d652d4" path="/var/lib/kubelet/pods/863dc71f-0017-4800-81a0-053404d652d4/volumes" Nov 04 01:28:42.054744 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:42.052275 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:43.223587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:43.218086 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:44.428503 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:44.428342 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:45.271141 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:45.269094 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:46.056269 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:46.056237 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:47.040920 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:47.040860 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:48.218449 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:48.215365 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:48.952147 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:48.952117 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:49.120035 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:49.119997 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:49.288156 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:49.288104 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:49.480466 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:49.480434 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:50.360023 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:50.359992 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr"] Nov 04 01:28:50.365703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:50.365682 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr" Nov 04 01:28:50.381001 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:50.380981 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-df6da4c0-dockercfg-v29w4\"" Nov 04 01:28:50.385372 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:50.385349 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr"] Nov 04 01:28:50.386966 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:50.386949 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-df6da4c0\"" Nov 04 01:28:50.458662 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:50.458638 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edada0ad-6b72-442b-ade8-b4f02d71f86c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr\" (UID: \"edada0ad-6b72-442b-ade8-b4f02d71f86c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr" Nov 04 01:28:50.458805 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:50.458775 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltm66\" (UniqueName: \"kubernetes.io/projected/edada0ad-6b72-442b-ade8-b4f02d71f86c-kube-api-access-ltm66\") pod \"payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr\" (UID: \"edada0ad-6b72-442b-ade8-b4f02d71f86c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr" Nov 04 01:28:50.575957 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:50.575914 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edada0ad-6b72-442b-ade8-b4f02d71f86c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr\" (UID: \"edada0ad-6b72-442b-ade8-b4f02d71f86c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr" Nov 04 01:28:50.576352 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:50.576325 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltm66\" (UniqueName: \"kubernetes.io/projected/edada0ad-6b72-442b-ade8-b4f02d71f86c-kube-api-access-ltm66\") pod \"payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr\" (UID: \"edada0ad-6b72-442b-ade8-b4f02d71f86c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr" Nov 04 01:28:50.589587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:50.578097 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edada0ad-6b72-442b-ade8-b4f02d71f86c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr\" (UID: \"edada0ad-6b72-442b-ade8-b4f02d71f86c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr" Nov 04 01:28:50.660641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:50.647241 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltm66\" (UniqueName: \"kubernetes.io/projected/edada0ad-6b72-442b-ade8-b4f02d71f86c-kube-api-access-ltm66\") pod \"payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr\" (UID: \"edada0ad-6b72-442b-ade8-b4f02d71f86c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr" Nov 04 01:28:50.694591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:50.691879 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr" Nov 04 01:28:50.771887 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:50.771853 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:50.956437 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:50.956350 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:51.239991 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:28:51.239948 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedada0ad_6b72_442b_ade8_b4f02d71f86c.slice/crio-3100bda9b21be49d7d373abefd4edf003d8c1c1523643ce8e2d9dc10c01e57ef WatchSource:0}: Error finding container 3100bda9b21be49d7d373abefd4edf003d8c1c1523643ce8e2d9dc10c01e57ef: Status 404 returned error can't find the container with id 3100bda9b21be49d7d373abefd4edf003d8c1c1523643ce8e2d9dc10c01e57ef Nov 04 01:28:51.240120 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:51.240045 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr"] Nov 04 01:28:51.369903 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:51.369855 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:51.445999 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:51.445974 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc" Nov 04 01:28:51.561274 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:51.561248 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:51.739586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:51.727259 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:51.979489 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:51.979389 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr" event={"ID":"edada0ad-6b72-442b-ade8-b4f02d71f86c","Type":"ContainerStarted","Data":"c0cc71a0e0185f4cfacb1f9c7143c79dd959ed165f702f2f9f5cd7d28da9fc37"} Nov 04 01:28:51.979796 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:51.979773 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr" event={"ID":"edada0ad-6b72-442b-ade8-b4f02d71f86c","Type":"ContainerStarted","Data":"3100bda9b21be49d7d373abefd4edf003d8c1c1523643ce8e2d9dc10c01e57ef"} Nov 04 01:28:52.005595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:51.984728 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr" Nov 04 01:28:52.030953 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:52.027402 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr" podStartSLOduration=2.027381059 podStartE2EDuration="2.027381059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:52.023086927 +0000 UTC m=+1814.138197047" watchObservedRunningTime="2026-11-04 01:28:52.027381059 +0000 UTC m=+1814.142491180" Nov 04 01:28:52.051443 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:52.051414 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bea7a66-xhdkc"] Nov 04 01:28:52.051644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:52.051455 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bea7a66-xhdkc"] Nov 04 01:28:52.477594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:52.477032 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:52.660783 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:52.660746 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:52.860672 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:52.860638 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:53.486980 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:53.486946 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:53.996482 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:53.996446 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7609e7b-4627-4850-a0b0-0e72aa4ba39f" path="/var/lib/kubelet/pods/f7609e7b-4627-4850-a0b0-0e72aa4ba39f/volumes" Nov 04 01:28:54.852599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:54.845465 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:55.827587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:55.825603 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:56.721226 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:56.721191 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:57.503869 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:57.503832 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:58.253339 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:58.246407 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:58.957473 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:58.957435 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:59.644169 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:28:59.644117 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:28:59.867035 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:28:59.866995 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:29:00.431333 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:00.431293 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:01.237184 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:01.237046 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:01.943800 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:01.943770 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:02.095384 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:02.095347 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:02.954904 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:02.954819 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:03.112044 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:03.112011 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:03.956845 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:03.956752 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:04.134587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:04.133841 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr" Nov 04 01:29:04.913748 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:04.913686 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:05.658596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:05.657589 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:06.552830 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:06.551230 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:06.728093 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:06.728065 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:07.446770 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:07.440843 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:08.608598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:08.605949 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:09.415470 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:09.413756 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:09.673069 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:09.672985 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:10.063082 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:10.063031 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:10.414092 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:10.414009 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:10.738644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:10.738345 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:11.512857 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:11.512516 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:12.217252 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:12.217217 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:12.889813 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:12.889780 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:13.054909 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:13.054863 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:13.536588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:13.536428 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:14.232689 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:14.232653 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:14.487231 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:14.487150 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:14.974240 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:14.974204 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:15.167592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:15.160530 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:15.579381 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:15.579348 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:15.789413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:15.789385 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:16.026123 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:16.026089 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:16.400959 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:16.400885 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:16.928627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:16.928559 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:17.919754 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:17.918110 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:18.161826 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:18.161769 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:18.659609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:18.648551 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:19.201765 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:19.201733 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:19.389341 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:19.389305 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:19.552454 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:19.552422 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:19.832137 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:19.832063 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:20.396750 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.396717 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:20.432128 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.432090 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb"] Nov 04 01:29:20.437877 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.437854 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.478588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.476888 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-5f572a-openshift-default-dockercfg-sxbbb\"" Nov 04 01:29:20.485592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.483020 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb"] Nov 04 01:29:20.520621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.516251 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-workload-certs\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.520621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.516335 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-credential-socket\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.520621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.516386 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57537b5d-473c-41da-9e67-6c848a66fee7-istio-token\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.520621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.516496 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-istio-data\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.529842 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.529807 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-istio-envoy\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.530194 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.530117 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-workload-socket\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.530487 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.530449 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t76wp\" (UniqueName: \"kubernetes.io/projected/57537b5d-473c-41da-9e67-6c848a66fee7-kube-api-access-t76wp\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.530733 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.530713 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57537b5d-473c-41da-9e67-6c848a66fee7-istiod-ca-cert\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.530930 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.530900 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57537b5d-473c-41da-9e67-6c848a66fee7-istio-podinfo\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.694642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.653207 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57537b5d-473c-41da-9e67-6c848a66fee7-istio-token\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.709682 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.709643 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-istio-data\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.710384 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.710358 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-istio-envoy\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.710949 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.710931 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-workload-socket\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.711539 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.711515 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t76wp\" (UniqueName: \"kubernetes.io/projected/57537b5d-473c-41da-9e67-6c848a66fee7-kube-api-access-t76wp\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.717171 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.717149 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57537b5d-473c-41da-9e67-6c848a66fee7-istiod-ca-cert\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.718041 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.718021 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57537b5d-473c-41da-9e67-6c848a66fee7-istio-podinfo\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.718606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.718586 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-workload-certs\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.718802 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.718786 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-credential-socket\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.719040 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.717927 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57537b5d-473c-41da-9e67-6c848a66fee7-istiod-ca-cert\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.719147 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.711320 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-workload-socket\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.719239 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.705524 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:20.723679 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.710242 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-istio-data\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.724164 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.724144 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-workload-certs\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.724785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.724732 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-credential-socket\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.732184 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.732158 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t76wp\" (UniqueName: \"kubernetes.io/projected/57537b5d-473c-41da-9e67-6c848a66fee7-kube-api-access-t76wp\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.886599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.884125 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57537b5d-473c-41da-9e67-6c848a66fee7-istio-token\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.887167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.887133 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-istio-envoy\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:20.925929 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:20.925755 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57537b5d-473c-41da-9e67-6c848a66fee7-istio-podinfo\") pod \"e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:21.015591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:21.011929 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:21.139199 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:21.138888 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:21.139902 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:21.139870 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:29:21.501638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:21.495092 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:21.760848 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:21.760798 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:21.937073 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:21.937045 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb"] Nov 04 01:29:21.938526 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:29:21.938491 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57537b5d_473c_41da_9e67_6c848a66fee7.slice/crio-385cc8d26d87915838ec18b101443ffdd81d28139ca3881531c8dedf36974e9c WatchSource:0}: Error finding container 385cc8d26d87915838ec18b101443ffdd81d28139ca3881531c8dedf36974e9c: Status 404 returned error can't find the container with id 385cc8d26d87915838ec18b101443ffdd81d28139ca3881531c8dedf36974e9c Nov 04 01:29:21.942214 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:21.942171 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:29:21.942291 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:21.942265 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:29:21.942333 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:21.942311 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:29:22.038943 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:22.038916 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" event={"ID":"57537b5d-473c-41da-9e67-6c848a66fee7","Type":"ContainerStarted","Data":"0205cb769edf06deebde37199e8ff5561c058fc76dbd456b0b36b62c04e5360e"} Nov 04 01:29:22.039120 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:22.038988 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" event={"ID":"57537b5d-473c-41da-9e67-6c848a66fee7","Type":"ContainerStarted","Data":"385cc8d26d87915838ec18b101443ffdd81d28139ca3881531c8dedf36974e9c"} Nov 04 01:29:22.098357 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:22.098292 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" podStartSLOduration=2.098272925 podStartE2EDuration="2.098272925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:22.093921831 +0000 UTC m=+1844.209031951" watchObservedRunningTime="2026-11-04 01:29:22.098272925 +0000 UTC m=+1844.213383044" Nov 04 01:29:22.140484 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:22.140455 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:22.646088 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:22.646011 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:23.147933 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:23.147905 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:23.831668 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:23.831590 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:24.297667 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:24.295960 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:24.298626 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:24.298600 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:29:24.955715 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:24.955666 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 04 01:29:25.059597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:25.058932 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:25.432682 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:25.432630 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:25.720051 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:25.719953 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:25.968389 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:25.968355 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:26.383591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:26.381074 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s"] Nov 04 01:29:26.571832 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:26.571790 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:26.893586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:26.872215 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:27.259966 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:27.259819 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:27.542653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:27.328813 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:27.542653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:27.451056 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:29:27.542653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:27.507752 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:28.422098 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:28.422030 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:29.240086 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:29.240048 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:29.897701 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:29:29.897662 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:29:29.940976 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:29.940951 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:30.093634 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:30.089150 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:30.714779 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:30.714732 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:31.519793 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:31.519759 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:32.298536 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:32.298503 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:33.136590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:33.127348 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:33.928597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:33.927722 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:34.631026 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:34.630898 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:35.605380 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:35.605344 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:36.460602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:36.460549 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:37.305455 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:37.305278 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:37.456232 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:37.456196 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:37.788906 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:37.788874 4852 generic.go:356] "Generic (PLEG): container finished" podID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerID="25d8e1ca785c573a8219c4cfff8b56925602c99c8b31a7953c1d3f17e234ac6e" exitCode=0 Nov 04 01:29:37.789451 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:37.788950 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" event={"ID":"c72aca29-e3f2-4a8f-abea-8841337cc569","Type":"ContainerDied","Data":"25d8e1ca785c573a8219c4cfff8b56925602c99c8b31a7953c1d3f17e234ac6e"} Nov 04 01:29:37.789659 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:37.789639 4852 scope.go:122] "RemoveContainer" containerID="25d8e1ca785c573a8219c4cfff8b56925602c99c8b31a7953c1d3f17e234ac6e" Nov 04 01:29:37.791542 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:37.791507 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:29:37.791659 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:37.791608 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:29:37.791659 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:37.791646 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:29:37.822737 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:37.822718 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:38.262596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:38.256083 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:38.403694 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:38.403668 4852 scope.go:122] "RemoveContainer" containerID="baf96c31a392bdddcfab9cbbf1d1523c3bbedf9466a6f46beb26a5400d2319ee" Nov 04 01:29:38.798751 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:38.798718 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" event={"ID":"c72aca29-e3f2-4a8f-abea-8841337cc569","Type":"ContainerStarted","Data":"a702664a3fa1c8f8d72c93af6783b851dd5ce5b09759f0922c7efc0a012b5979"} Nov 04 01:29:39.154194 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:39.152612 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:39.774878 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:39.774845 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:29:39.783080 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:39.783050 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:29:39.783229 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:39.783105 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:29:40.094107 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:40.094024 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:40.265842 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:40.265747 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:40.764040 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:40.764001 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:29:40.764348 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:40.764318 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:29:41.191175 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:41.191085 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:41.551371 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:41.551334 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" containerID="cri-o://be820ff88fc70e0765ae7c9f42b07cf26d2db5e006e061512174a0d3263d189a" gracePeriod=45 Nov 04 01:29:41.592298 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:41.591588 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": dial tcp 10.134.0.33:8000: connect: connection refused" Nov 04 01:29:41.787905 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:41.768054 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:29:41.787905 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:41.768116 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:29:41.926344 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:41.926167 4852 generic.go:356] "Generic (PLEG): container finished" podID="abe48140-3cf1-43a8-a814-18f86e309795" containerID="be820ff88fc70e0765ae7c9f42b07cf26d2db5e006e061512174a0d3263d189a" exitCode=0 Nov 04 01:29:41.926344 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:41.926231 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" event={"ID":"abe48140-3cf1-43a8-a814-18f86e309795","Type":"ContainerDied","Data":"be820ff88fc70e0765ae7c9f42b07cf26d2db5e006e061512174a0d3263d189a"} Nov 04 01:29:42.052312 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.052279 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:29:42.089136 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.089098 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:42.214445 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.214362 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-model-cache\") pod \"abe48140-3cf1-43a8-a814-18f86e309795\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " Nov 04 01:29:42.214917 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.214501 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-dshm\" (UniqueName: \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-dshm\") pod \"abe48140-3cf1-43a8-a814-18f86e309795\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " Nov 04 01:29:42.214917 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.214540 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abe48140-3cf1-43a8-a814-18f86e309795-kube-api-access-8csqp\" (UniqueName: \"kubernetes.io/projected/abe48140-3cf1-43a8-a814-18f86e309795-kube-api-access-8csqp\") pod \"abe48140-3cf1-43a8-a814-18f86e309795\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " Nov 04 01:29:42.214917 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.214635 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abe48140-3cf1-43a8-a814-18f86e309795-tls-certs\" (UniqueName: \"kubernetes.io/secret/abe48140-3cf1-43a8-a814-18f86e309795-tls-certs\") pod \"abe48140-3cf1-43a8-a814-18f86e309795\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " Nov 04 01:29:42.214917 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.214662 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-model-cache" pod "abe48140-3cf1-43a8-a814-18f86e309795" (UID: "abe48140-3cf1-43a8-a814-18f86e309795"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:42.214917 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.214727 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-home\" (UniqueName: \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-home\") pod \"abe48140-3cf1-43a8-a814-18f86e309795\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " Nov 04 01:29:42.214917 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.214779 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-tmp-dir\") pod \"abe48140-3cf1-43a8-a814-18f86e309795\" (UID: \"abe48140-3cf1-43a8-a814-18f86e309795\") " Nov 04 01:29:42.215291 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.215029 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-home" pod "abe48140-3cf1-43a8-a814-18f86e309795" (UID: "abe48140-3cf1-43a8-a814-18f86e309795"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:42.215291 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.215111 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-tmp-dir" pod "abe48140-3cf1-43a8-a814-18f86e309795" (UID: "abe48140-3cf1-43a8-a814-18f86e309795"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:42.215911 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.215886 4852 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-home\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:42.216029 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.215917 4852 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-tmp-dir\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:42.216029 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.215932 4852 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-model-cache\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:42.217060 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.217037 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-dshm" pod "abe48140-3cf1-43a8-a814-18f86e309795" (UID: "abe48140-3cf1-43a8-a814-18f86e309795"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:42.217155 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.217089 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abe48140-3cf1-43a8-a814-18f86e309795-tls-certs" pod "abe48140-3cf1-43a8-a814-18f86e309795" (UID: "abe48140-3cf1-43a8-a814-18f86e309795"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:29:42.217155 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.217136 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe48140-3cf1-43a8-a814-18f86e309795-kube-api-access-8csqp" pod "abe48140-3cf1-43a8-a814-18f86e309795" (UID: "abe48140-3cf1-43a8-a814-18f86e309795"). InnerVolumeSpecName "kube-api-access-8csqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:42.301256 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.301034 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:42.324602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.318494 4852 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abe48140-3cf1-43a8-a814-18f86e309795-dshm\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:42.324602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.318531 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8csqp\" (UniqueName: \"kubernetes.io/projected/abe48140-3cf1-43a8-a814-18f86e309795-kube-api-access-8csqp\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:42.324602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.318547 4852 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe48140-3cf1-43a8-a814-18f86e309795-tls-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:42.777271 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.776893 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:29:42.777271 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.776964 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:29:42.934408 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.934383 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" Nov 04 01:29:42.934627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.934418 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s" event={"ID":"abe48140-3cf1-43a8-a814-18f86e309795","Type":"ContainerDied","Data":"5b6ea630e32e3b2f4abc7538fc12b1972ae696fe5ac056765af0bd85b9b70268"} Nov 04 01:29:42.934627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:42.934483 4852 scope.go:122] "RemoveContainer" containerID="be820ff88fc70e0765ae7c9f42b07cf26d2db5e006e061512174a0d3263d189a" Nov 04 01:29:43.001908 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:43.001874 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s"] Nov 04 01:29:43.004785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:43.004759 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/simulator-w0-ee876a-kserve-74549bbf9c-h2g6s"] Nov 04 01:29:43.218332 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:43.218226 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:43.762589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:43.760888 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:29:43.762589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:43.760946 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:29:43.767586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:43.766646 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:29:43.794695 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:43.794668 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd"] Nov 04 01:29:43.965404 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:43.965355 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:43.998891 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:43.998865 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abe48140-3cf1-43a8-a814-18f86e309795" path="/var/lib/kubelet/pods/abe48140-3cf1-43a8-a814-18f86e309795/volumes" Nov 04 01:29:44.160139 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:44.160053 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:44.761590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:44.760847 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:29:44.761590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:44.760908 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 04 01:29:44.828752 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:44.828727 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:44.992502 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:44.992469 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:45.670115 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:45.670078 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:45.766887 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:45.766860 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:29:45.767931 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:45.767910 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:29:46.350538 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:46.350503 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:46.498102 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:46.498026 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:46.816295 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:46.816264 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524"] Nov 04 01:29:46.817086 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:46.817063 4852 cpu_manager.go:414] "containerMap: removing container" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" Nov 04 01:29:46.817183 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:46.817091 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" Nov 04 01:29:46.817307 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:46.817293 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="abe48140-3cf1-43a8-a814-18f86e309795" containerName="main" Nov 04 01:29:46.820711 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:46.820691 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:46.839337 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:46.839309 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524"] Nov 04 01:29:46.842921 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:46.842897 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-df6da4c0-openshift-default-dockercfg-78n5r\"" Nov 04 01:29:46.940939 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:46.940897 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:46.941083 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:46.940955 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:46.941137 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:46.941111 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:46.941178 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:46.941138 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:46.941245 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:46.941232 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158b1659-369b-454e-bb71-db3763ca6774-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:46.941300 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:46.941287 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:46.941378 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:46.941367 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158b1659-369b-454e-bb71-db3763ca6774-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:46.941448 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:46.941428 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpqdj\" (UniqueName: \"kubernetes.io/projected/158b1659-369b-454e-bb71-db3763ca6774-kube-api-access-gpqdj\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:46.941514 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:46.941498 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158b1659-369b-454e-bb71-db3763ca6774-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:47.043319 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.043292 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:47.043471 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.043385 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158b1659-369b-454e-bb71-db3763ca6774-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:47.043471 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.043426 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpqdj\" (UniqueName: \"kubernetes.io/projected/158b1659-369b-454e-bb71-db3763ca6774-kube-api-access-gpqdj\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:47.043471 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.043458 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158b1659-369b-454e-bb71-db3763ca6774-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:47.043644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.043499 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:47.043644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.043538 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:47.043756 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.043713 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:47.043756 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.043735 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:47.043866 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.043847 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158b1659-369b-454e-bb71-db3763ca6774-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:47.044006 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.043966 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:47.044301 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.044075 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:47.044301 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.044131 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:47.044301 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.044160 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:47.044496 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.044429 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158b1659-369b-454e-bb71-db3763ca6774-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:47.045973 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.045949 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:47.046069 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.046036 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158b1659-369b-454e-bb71-db3763ca6774-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:47.062342 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.062312 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpqdj\" (UniqueName: \"kubernetes.io/projected/158b1659-369b-454e-bb71-db3763ca6774-kube-api-access-gpqdj\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:47.062438 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.062350 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158b1659-369b-454e-bb71-db3763ca6774-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:47.150584 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.149846 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:47.194419 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.194294 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:47.523741 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.523713 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524"] Nov 04 01:29:47.523891 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:29:47.523872 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158b1659_369b_454e_bb71_db3763ca6774.slice/crio-2d626a7eea6878f3c75ec0c6c6ef97b7d6ea8625af9f77a234beaf7e7b707ed7 WatchSource:0}: Error finding container 2d626a7eea6878f3c75ec0c6c6ef97b7d6ea8625af9f77a234beaf7e7b707ed7: Status 404 returned error can't find the container with id 2d626a7eea6878f3c75ec0c6c6ef97b7d6ea8625af9f77a234beaf7e7b707ed7 Nov 04 01:29:47.526624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.526585 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:29:47.526754 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.526660 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:29:47.526754 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:47.526691 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:29:48.011643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:48.011607 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" event={"ID":"158b1659-369b-454e-bb71-db3763ca6774","Type":"ContainerStarted","Data":"666b7511555868d7af6e2fcefbb8cecb2463ae9df141e161879443c14cf4278a"} Nov 04 01:29:48.012018 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:48.011653 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" event={"ID":"158b1659-369b-454e-bb71-db3763ca6774","Type":"ContainerStarted","Data":"2d626a7eea6878f3c75ec0c6c6ef97b7d6ea8625af9f77a234beaf7e7b707ed7"} Nov 04 01:29:48.065762 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:48.065719 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" podStartSLOduration=2.065706885 podStartE2EDuration="2.065706885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:48.064932604 +0000 UTC m=+1870.180042721" watchObservedRunningTime="2026-11-04 01:29:48.065706885 +0000 UTC m=+1870.180817058" Nov 04 01:29:48.150646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:48.150620 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:48.151887 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:48.151865 4852 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 04 01:29:48.151999 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:48.151908 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" podUID="158b1659-369b-454e-bb71-db3763ca6774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 04 01:29:48.286648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:48.286489 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:48.795831 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:48.795785 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd" podUID="38bc84cb-29d4-4979-8d64-17393f82c169" containerName="payload-processing" containerID="cri-o://ad53f09f2ee93b18a9a3a3deaee7b34c34235f63b5903d787886e5373a3191ad" gracePeriod=25 Nov 04 01:29:49.022533 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:49.022503 4852 generic.go:356] "Generic (PLEG): container finished" podID="38bc84cb-29d4-4979-8d64-17393f82c169" containerID="ad53f09f2ee93b18a9a3a3deaee7b34c34235f63b5903d787886e5373a3191ad" exitCode=0 Nov 04 01:29:49.022998 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:49.022595 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd" event={"ID":"38bc84cb-29d4-4979-8d64-17393f82c169","Type":"ContainerDied","Data":"ad53f09f2ee93b18a9a3a3deaee7b34c34235f63b5903d787886e5373a3191ad"} Nov 04 01:29:49.118071 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:49.118042 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd" Nov 04 01:29:49.124757 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:49.124737 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38bc84cb-29d4-4979-8d64-17393f82c169-kube-api-access-ffgtx\" (UniqueName: \"kubernetes.io/projected/38bc84cb-29d4-4979-8d64-17393f82c169-kube-api-access-ffgtx\") pod \"38bc84cb-29d4-4979-8d64-17393f82c169\" (UID: \"38bc84cb-29d4-4979-8d64-17393f82c169\") " Nov 04 01:29:49.124889 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:49.124877 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38bc84cb-29d4-4979-8d64-17393f82c169-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38bc84cb-29d4-4979-8d64-17393f82c169-plugins-config-volume\") pod \"38bc84cb-29d4-4979-8d64-17393f82c169\" (UID: \"38bc84cb-29d4-4979-8d64-17393f82c169\") " Nov 04 01:29:49.125241 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:49.125214 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38bc84cb-29d4-4979-8d64-17393f82c169-plugins-config-volume" pod "38bc84cb-29d4-4979-8d64-17393f82c169" (UID: "38bc84cb-29d4-4979-8d64-17393f82c169"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:29:49.125725 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:49.125703 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38bc84cb-29d4-4979-8d64-17393f82c169-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:49.127032 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:49.127007 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38bc84cb-29d4-4979-8d64-17393f82c169-kube-api-access-ffgtx" pod "38bc84cb-29d4-4979-8d64-17393f82c169" (UID: "38bc84cb-29d4-4979-8d64-17393f82c169"). InnerVolumeSpecName "kube-api-access-ffgtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:49.263590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:49.263352 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffgtx\" (UniqueName: \"kubernetes.io/projected/38bc84cb-29d4-4979-8d64-17393f82c169-kube-api-access-ffgtx\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:49.325591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:49.324791 4852 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 04 01:29:49.329912 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:49.329820 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:49.330820 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:49.330542 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" podUID="158b1659-369b-454e-bb71-db3763ca6774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 04 01:29:49.820950 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:49.820920 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:50.052829 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:50.052791 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:50.142947 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:50.142865 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd" Nov 04 01:29:50.147476 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:50.147188 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd" event={"ID":"38bc84cb-29d4-4979-8d64-17393f82c169","Type":"ContainerDied","Data":"a984f3bf7778a44ff31b9f8afd38d20c3618f817b796384cd9e9256683d3861f"} Nov 04 01:29:50.147476 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:50.147284 4852 scope.go:122] "RemoveContainer" containerID="ad53f09f2ee93b18a9a3a3deaee7b34c34235f63b5903d787886e5373a3191ad" Nov 04 01:29:50.150446 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:50.150312 4852 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 04 01:29:50.150446 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:50.150361 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" podUID="158b1659-369b-454e-bb71-db3763ca6774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 04 01:29:50.203721 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:50.203692 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd"] Nov 04 01:29:50.207897 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:50.207877 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ee876a-6659594494-sb4vd"] Nov 04 01:29:50.715960 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:50.715931 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:51.150353 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:51.150323 4852 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 04 01:29:51.150777 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:51.150372 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" podUID="158b1659-369b-454e-bb71-db3763ca6774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 04 01:29:51.662590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:51.662342 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:51.997130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:51.997048 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38bc84cb-29d4-4979-8d64-17393f82c169" path="/var/lib/kubelet/pods/38bc84cb-29d4-4979-8d64-17393f82c169/volumes" Nov 04 01:29:52.150169 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:52.150140 4852 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 04 01:29:52.150346 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:52.150187 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" podUID="158b1659-369b-454e-bb71-db3763ca6774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 04 01:29:52.199726 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:52.199701 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:52.667300 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:52.667206 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:52.843020 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:52.842987 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4"] Nov 04 01:29:52.843707 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:52.843692 4852 cpu_manager.go:414] "containerMap: removing container" podUID="38bc84cb-29d4-4979-8d64-17393f82c169" containerName="payload-processing" Nov 04 01:29:52.843707 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:52.843707 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38bc84cb-29d4-4979-8d64-17393f82c169" containerName="payload-processing" Nov 04 01:29:52.843903 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:52.843891 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="38bc84cb-29d4-4979-8d64-17393f82c169" containerName="payload-processing" Nov 04 01:29:52.846396 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:52.846376 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:52.851151 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:52.850986 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-75f7ae-openshift-default-dockercfg-v4pjw\"" Nov 04 01:29:52.857091 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:52.857049 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4"] Nov 04 01:29:52.903130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:52.903101 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-data\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:52.903346 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:52.903330 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istiod-ca-cert\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:52.903556 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:52.903541 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-workload-certs\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:52.903690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:52.903669 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn9z8\" (UniqueName: \"kubernetes.io/projected/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-kube-api-access-fn9z8\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:52.903764 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:52.903751 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-workload-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:52.903817 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:52.903784 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-envoy\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:52.903915 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:52.903900 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-credential-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:52.904002 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:52.903990 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-token\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:52.904094 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:52.904080 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-podinfo\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:53.005889 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.005856 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-token\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:53.006054 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.005939 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-podinfo\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:53.006054 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.005991 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-data\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:53.006175 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.006066 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istiod-ca-cert\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:53.006175 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.006129 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-workload-certs\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:53.006277 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.006186 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn9z8\" (UniqueName: \"kubernetes.io/projected/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-kube-api-access-fn9z8\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:53.006277 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.006254 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-workload-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:53.006380 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.006281 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-envoy\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:53.006437 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.006404 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-credential-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:53.006540 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.006476 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-data\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:53.006796 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.006777 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-credential-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:53.006883 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.006864 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-workload-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:53.006969 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.006946 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istiod-ca-cert\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:53.007286 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.007266 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-workload-certs\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:53.008882 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.008858 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-envoy\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:53.009275 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.009259 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-podinfo\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:53.013955 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.013934 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-token\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:53.014283 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.014264 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn9z8\" (UniqueName: \"kubernetes.io/projected/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-kube-api-access-fn9z8\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:53.082475 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.082429 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:53.150324 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.150297 4852 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 04 01:29:53.150480 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.150352 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" podUID="158b1659-369b-454e-bb71-db3763ca6774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 04 01:29:53.190375 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.190346 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:53.527774 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.527197 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:53.539905 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:29:53.532544 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af6084e_6b1b_4728_b4d0_6ca48db7a30b.slice/crio-09d12b02e9f2a5b5f50d8b7c8271edcd5a7ab151a059fa84342ca24117dc8e10 WatchSource:0}: Error finding container 09d12b02e9f2a5b5f50d8b7c8271edcd5a7ab151a059fa84342ca24117dc8e10: Status 404 returned error can't find the container with id 09d12b02e9f2a5b5f50d8b7c8271edcd5a7ab151a059fa84342ca24117dc8e10 Nov 04 01:29:53.539905 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.538409 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4"] Nov 04 01:29:53.583442 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.571975 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:29:53.583442 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.572073 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:29:53.583442 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:53.572127 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:29:54.196737 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:54.196662 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" event={"ID":"2af6084e-6b1b-4728-b4d0-6ca48db7a30b","Type":"ContainerStarted","Data":"ab6f6b48bcc284ffd00135ad9b5508d9f3ac4400cb370d99c9536c2ee972854a"} Nov 04 01:29:54.196940 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:54.196860 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" event={"ID":"2af6084e-6b1b-4728-b4d0-6ca48db7a30b","Type":"ContainerStarted","Data":"09d12b02e9f2a5b5f50d8b7c8271edcd5a7ab151a059fa84342ca24117dc8e10"} Nov 04 01:29:54.198347 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:54.198315 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:54.198972 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:54.198944 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:54.199997 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:54.199976 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:29:54.225507 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:54.225163 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" podStartSLOduration=2.225145225 podStartE2EDuration="2.225145225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:54.224507113 +0000 UTC m=+1876.339617230" watchObservedRunningTime="2026-11-04 01:29:54.225145225 +0000 UTC m=+1876.340255343" Nov 04 01:29:54.484440 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:54.456881 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:55.191071 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:55.191037 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:29:55.193127 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:55.193101 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 04 01:29:55.193271 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:55.193150 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" podUID="2af6084e-6b1b-4728-b4d0-6ca48db7a30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 04 01:29:55.405325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:55.405190 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:56.197599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:56.194049 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 04 01:29:56.197599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:56.194120 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" podUID="2af6084e-6b1b-4728-b4d0-6ca48db7a30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 04 01:29:56.513670 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:56.507926 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:57.190834 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:57.190801 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 04 01:29:57.191031 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:57.190861 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" podUID="2af6084e-6b1b-4728-b4d0-6ca48db7a30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 04 01:29:57.373623 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:57.365189 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:58.191591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:58.185631 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:58.191591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:58.190720 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 04 01:29:58.191591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:58.190790 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" podUID="2af6084e-6b1b-4728-b4d0-6ca48db7a30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 04 01:29:59.068060 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:59.068022 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:59.191250 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:59.191217 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 04 01:29:59.191510 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:59.191282 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" podUID="2af6084e-6b1b-4728-b4d0-6ca48db7a30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 04 01:29:59.293728 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:59.293695 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:59.880586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:29:59.878682 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:29:59.998263 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:29:59.998225 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:30:00.191590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.191479 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 04 01:30:00.191590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.191578 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" podUID="2af6084e-6b1b-4728-b4d0-6ca48db7a30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 04 01:30:00.238630 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.238501 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh"] Nov 04 01:30:00.245333 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.245304 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh" Nov 04 01:30:00.258543 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.258507 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh"] Nov 04 01:30:00.261412 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.261389 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:30:00.274094 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.274063 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n"] Nov 04 01:30:00.279656 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.279620 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n" Nov 04 01:30:00.308462 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.308437 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n"] Nov 04 01:30:00.328344 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.328297 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:00.355030 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.354998 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4"] Nov 04 01:30:00.361326 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.361299 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4" Nov 04 01:30:00.368079 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.367279 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8"] Nov 04 01:30:00.377477 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.377458 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n"] Nov 04 01:30:00.378768 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.378222 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8" Nov 04 01:30:00.384637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.382905 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f011e7-dc3c-4b35-8c59-76bd4101237c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n\" (UID: \"c1f011e7-dc3c-4b35-8c59-76bd4101237c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n" Nov 04 01:30:00.384637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.382988 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcd5h\" (UniqueName: \"kubernetes.io/projected/c1f011e7-dc3c-4b35-8c59-76bd4101237c-kube-api-access-vcd5h\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n\" (UID: \"c1f011e7-dc3c-4b35-8c59-76bd4101237c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n" Nov 04 01:30:00.384637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.383098 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48244ae6-4584-4874-b776-610df53ed021-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh\" (UID: \"48244ae6-4584-4874-b776-610df53ed021\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh" Nov 04 01:30:00.384637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.383188 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm6xj\" (UniqueName: \"kubernetes.io/projected/48244ae6-4584-4874-b776-610df53ed021-kube-api-access-rm6xj\") pod \"maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh\" (UID: \"48244ae6-4584-4874-b776-610df53ed021\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh" Nov 04 01:30:00.385943 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.385760 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n" Nov 04 01:30:00.399802 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.399778 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n"] Nov 04 01:30:00.406294 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.406276 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8"] Nov 04 01:30:00.410482 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.410453 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4"] Nov 04 01:30:00.441316 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.441281 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7"] Nov 04 01:30:00.456851 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.456777 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9"] Nov 04 01:30:00.457845 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.457815 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7" Nov 04 01:30:00.466110 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.466088 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5"] Nov 04 01:30:00.467736 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.467679 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9" Nov 04 01:30:00.491798 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.487677 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a17cae9-e3d0-4a80-93ba-5602baba0d74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n\" (UID: \"4a17cae9-e3d0-4a80-93ba-5602baba0d74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n" Nov 04 01:30:00.491798 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.487721 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrfzc\" (UniqueName: \"kubernetes.io/projected/cb73d5de-5e15-4453-80fc-e643b3dbd961-kube-api-access-qrfzc\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8\" (UID: \"cb73d5de-5e15-4453-80fc-e643b3dbd961\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8" Nov 04 01:30:00.491798 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.487789 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f011e7-dc3c-4b35-8c59-76bd4101237c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n\" (UID: \"c1f011e7-dc3c-4b35-8c59-76bd4101237c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n" Nov 04 01:30:00.492102 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.492023 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcd5h\" (UniqueName: \"kubernetes.io/projected/c1f011e7-dc3c-4b35-8c59-76bd4101237c-kube-api-access-vcd5h\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n\" (UID: \"c1f011e7-dc3c-4b35-8c59-76bd4101237c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n" Nov 04 01:30:00.492102 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.492070 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc68a724-a4bd-436b-92ff-4433c18960f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4\" (UID: \"dc68a724-a4bd-436b-92ff-4433c18960f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4" Nov 04 01:30:00.492381 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.492269 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48244ae6-4584-4874-b776-610df53ed021-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh\" (UID: \"48244ae6-4584-4874-b776-610df53ed021\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh" Nov 04 01:30:00.492381 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.492320 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb73d5de-5e15-4453-80fc-e643b3dbd961-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8\" (UID: \"cb73d5de-5e15-4453-80fc-e643b3dbd961\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8" Nov 04 01:30:00.492381 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.492352 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59c7b\" (UniqueName: \"kubernetes.io/projected/dc68a724-a4bd-436b-92ff-4433c18960f2-kube-api-access-59c7b\") pod \"maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4\" (UID: \"dc68a724-a4bd-436b-92ff-4433c18960f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4" Nov 04 01:30:00.492623 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.492426 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm6xj\" (UniqueName: \"kubernetes.io/projected/48244ae6-4584-4874-b776-610df53ed021-kube-api-access-rm6xj\") pod \"maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh\" (UID: \"48244ae6-4584-4874-b776-610df53ed021\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh" Nov 04 01:30:00.492683 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.492652 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drdsq\" (UniqueName: \"kubernetes.io/projected/4a17cae9-e3d0-4a80-93ba-5602baba0d74-kube-api-access-drdsq\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n\" (UID: \"4a17cae9-e3d0-4a80-93ba-5602baba0d74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n" Nov 04 01:30:00.501167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.494157 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f011e7-dc3c-4b35-8c59-76bd4101237c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n\" (UID: \"c1f011e7-dc3c-4b35-8c59-76bd4101237c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n" Nov 04 01:30:00.501167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.495224 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48244ae6-4584-4874-b776-610df53ed021-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh\" (UID: \"48244ae6-4584-4874-b776-610df53ed021\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh" Nov 04 01:30:00.501167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.499470 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9"] Nov 04 01:30:00.501167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.499509 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5"] Nov 04 01:30:00.501167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.499624 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5" Nov 04 01:30:00.510801 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.510772 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7"] Nov 04 01:30:00.511275 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.511254 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcd5h\" (UniqueName: \"kubernetes.io/projected/c1f011e7-dc3c-4b35-8c59-76bd4101237c-kube-api-access-vcd5h\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n\" (UID: \"c1f011e7-dc3c-4b35-8c59-76bd4101237c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n" Nov 04 01:30:00.514299 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.514273 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm6xj\" (UniqueName: \"kubernetes.io/projected/48244ae6-4584-4874-b776-610df53ed021-kube-api-access-rm6xj\") pod \"maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh\" (UID: \"48244ae6-4584-4874-b776-610df53ed021\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh" Nov 04 01:30:00.570088 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.570057 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh" Nov 04 01:30:00.595354 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.595314 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2788c94-d82f-4ac6-94db-71ff7aeec55d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7\" (UID: \"b2788c94-d82f-4ac6-94db-71ff7aeec55d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7" Nov 04 01:30:00.595517 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.595422 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc68a724-a4bd-436b-92ff-4433c18960f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4\" (UID: \"dc68a724-a4bd-436b-92ff-4433c18960f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4" Nov 04 01:30:00.596190 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.596163 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc68a724-a4bd-436b-92ff-4433c18960f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4\" (UID: \"dc68a724-a4bd-436b-92ff-4433c18960f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4" Nov 04 01:30:00.596336 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.596225 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f61cb5-7acc-4d99-9b95-2c7492876e68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5\" (UID: \"03f61cb5-7acc-4d99-9b95-2c7492876e68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5" Nov 04 01:30:00.596336 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.596297 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9585268-ca61-4539-84d5-a986d3161a1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9\" (UID: \"d9585268-ca61-4539-84d5-a986d3161a1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9" Nov 04 01:30:00.596492 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.596390 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkfs7\" (UniqueName: \"kubernetes.io/projected/b2788c94-d82f-4ac6-94db-71ff7aeec55d-kube-api-access-qkfs7\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7\" (UID: \"b2788c94-d82f-4ac6-94db-71ff7aeec55d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7" Nov 04 01:30:00.596492 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.596433 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb73d5de-5e15-4453-80fc-e643b3dbd961-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8\" (UID: \"cb73d5de-5e15-4453-80fc-e643b3dbd961\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8" Nov 04 01:30:00.596492 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.596466 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59c7b\" (UniqueName: \"kubernetes.io/projected/dc68a724-a4bd-436b-92ff-4433c18960f2-kube-api-access-59c7b\") pod \"maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4\" (UID: \"dc68a724-a4bd-436b-92ff-4433c18960f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4" Nov 04 01:30:00.596680 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.596538 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdpzp\" (UniqueName: \"kubernetes.io/projected/d9585268-ca61-4539-84d5-a986d3161a1d-kube-api-access-qdpzp\") pod \"maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9\" (UID: \"d9585268-ca61-4539-84d5-a986d3161a1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9" Nov 04 01:30:00.596744 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.596733 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drdsq\" (UniqueName: \"kubernetes.io/projected/4a17cae9-e3d0-4a80-93ba-5602baba0d74-kube-api-access-drdsq\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n\" (UID: \"4a17cae9-e3d0-4a80-93ba-5602baba0d74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n" Nov 04 01:30:00.596874 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.596855 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgbvv\" (UniqueName: \"kubernetes.io/projected/03f61cb5-7acc-4d99-9b95-2c7492876e68-kube-api-access-sgbvv\") pod \"maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5\" (UID: \"03f61cb5-7acc-4d99-9b95-2c7492876e68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5" Nov 04 01:30:00.596950 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.596931 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a17cae9-e3d0-4a80-93ba-5602baba0d74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n\" (UID: \"4a17cae9-e3d0-4a80-93ba-5602baba0d74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n" Nov 04 01:30:00.597068 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.596962 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrfzc\" (UniqueName: \"kubernetes.io/projected/cb73d5de-5e15-4453-80fc-e643b3dbd961-kube-api-access-qrfzc\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8\" (UID: \"cb73d5de-5e15-4453-80fc-e643b3dbd961\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8" Nov 04 01:30:00.598085 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.598061 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb73d5de-5e15-4453-80fc-e643b3dbd961-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8\" (UID: \"cb73d5de-5e15-4453-80fc-e643b3dbd961\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8" Nov 04 01:30:00.606317 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.601557 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a17cae9-e3d0-4a80-93ba-5602baba0d74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n\" (UID: \"4a17cae9-e3d0-4a80-93ba-5602baba0d74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n" Nov 04 01:30:00.606317 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.602964 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n" Nov 04 01:30:00.609158 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.608953 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrfzc\" (UniqueName: \"kubernetes.io/projected/cb73d5de-5e15-4453-80fc-e643b3dbd961-kube-api-access-qrfzc\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8\" (UID: \"cb73d5de-5e15-4453-80fc-e643b3dbd961\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8" Nov 04 01:30:00.612017 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.611760 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59c7b\" (UniqueName: \"kubernetes.io/projected/dc68a724-a4bd-436b-92ff-4433c18960f2-kube-api-access-59c7b\") pod \"maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4\" (UID: \"dc68a724-a4bd-436b-92ff-4433c18960f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4" Nov 04 01:30:00.612985 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.612959 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drdsq\" (UniqueName: \"kubernetes.io/projected/4a17cae9-e3d0-4a80-93ba-5602baba0d74-kube-api-access-drdsq\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n\" (UID: \"4a17cae9-e3d0-4a80-93ba-5602baba0d74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n" Nov 04 01:30:00.685379 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.685129 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4" Nov 04 01:30:00.702940 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.701131 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdpzp\" (UniqueName: \"kubernetes.io/projected/d9585268-ca61-4539-84d5-a986d3161a1d-kube-api-access-qdpzp\") pod \"maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9\" (UID: \"d9585268-ca61-4539-84d5-a986d3161a1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9" Nov 04 01:30:00.702940 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.701384 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgbvv\" (UniqueName: \"kubernetes.io/projected/03f61cb5-7acc-4d99-9b95-2c7492876e68-kube-api-access-sgbvv\") pod \"maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5\" (UID: \"03f61cb5-7acc-4d99-9b95-2c7492876e68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5" Nov 04 01:30:00.702940 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.701508 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2788c94-d82f-4ac6-94db-71ff7aeec55d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7\" (UID: \"b2788c94-d82f-4ac6-94db-71ff7aeec55d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7" Nov 04 01:30:00.702940 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.701542 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f61cb5-7acc-4d99-9b95-2c7492876e68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5\" (UID: \"03f61cb5-7acc-4d99-9b95-2c7492876e68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5" Nov 04 01:30:00.702940 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.701592 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9585268-ca61-4539-84d5-a986d3161a1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9\" (UID: \"d9585268-ca61-4539-84d5-a986d3161a1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9" Nov 04 01:30:00.702940 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.701680 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkfs7\" (UniqueName: \"kubernetes.io/projected/b2788c94-d82f-4ac6-94db-71ff7aeec55d-kube-api-access-qkfs7\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7\" (UID: \"b2788c94-d82f-4ac6-94db-71ff7aeec55d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7" Nov 04 01:30:00.703988 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.703791 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2788c94-d82f-4ac6-94db-71ff7aeec55d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7\" (UID: \"b2788c94-d82f-4ac6-94db-71ff7aeec55d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7" Nov 04 01:30:00.704952 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.704448 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f61cb5-7acc-4d99-9b95-2c7492876e68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5\" (UID: \"03f61cb5-7acc-4d99-9b95-2c7492876e68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5" Nov 04 01:30:00.705904 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.705885 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9585268-ca61-4539-84d5-a986d3161a1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9\" (UID: \"d9585268-ca61-4539-84d5-a986d3161a1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9" Nov 04 01:30:00.715415 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.714640 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkfs7\" (UniqueName: \"kubernetes.io/projected/b2788c94-d82f-4ac6-94db-71ff7aeec55d-kube-api-access-qkfs7\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7\" (UID: \"b2788c94-d82f-4ac6-94db-71ff7aeec55d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7" Nov 04 01:30:00.715996 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.715964 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdpzp\" (UniqueName: \"kubernetes.io/projected/d9585268-ca61-4539-84d5-a986d3161a1d-kube-api-access-qdpzp\") pod \"maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9\" (UID: \"d9585268-ca61-4539-84d5-a986d3161a1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9" Nov 04 01:30:00.716878 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.716831 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgbvv\" (UniqueName: \"kubernetes.io/projected/03f61cb5-7acc-4d99-9b95-2c7492876e68-kube-api-access-sgbvv\") pod \"maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5\" (UID: \"03f61cb5-7acc-4d99-9b95-2c7492876e68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5" Nov 04 01:30:00.717681 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.717446 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8" Nov 04 01:30:00.754147 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.753904 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n" Nov 04 01:30:00.809630 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.809599 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7" Nov 04 01:30:00.817010 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.816800 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9" Nov 04 01:30:00.864257 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.864027 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5" Nov 04 01:30:00.889478 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.889116 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh"] Nov 04 01:30:00.937969 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.937931 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n"] Nov 04 01:30:00.997006 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:00.996982 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4"] Nov 04 01:30:01.067494 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.067462 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8"] Nov 04 01:30:01.081789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.081766 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n"] Nov 04 01:30:01.207799 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.205512 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9"] Nov 04 01:30:01.207799 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.207109 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:01.243231 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.241669 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:30:01.244620 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.243492 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:30:01.244753 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.244663 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:30:01.246272 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.246244 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5"] Nov 04 01:30:01.326885 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.326854 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7"] Nov 04 01:30:01.353978 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.351645 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4" event={"ID":"dc68a724-a4bd-436b-92ff-4433c18960f2","Type":"ContainerStarted","Data":"65d9011f22f6b90bc71ced6f053dfa76f19bb7573efac81d98868126c92951b9"} Nov 04 01:30:01.355941 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.355916 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9" event={"ID":"d9585268-ca61-4539-84d5-a986d3161a1d","Type":"ContainerStarted","Data":"efa0d13a7bd175f4f1ffe9a895c24be37da02aab9189106a49c26128ca06096e"} Nov 04 01:30:01.359339 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.359317 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8" event={"ID":"cb73d5de-5e15-4453-80fc-e643b3dbd961","Type":"ContainerStarted","Data":"18648782cd350c5ce9637d34b704cbd99b7afb1b136fb0edd6253bb796819e5e"} Nov 04 01:30:01.361805 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.361786 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:01.363724 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.363699 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5" event={"ID":"03f61cb5-7acc-4d99-9b95-2c7492876e68","Type":"ContainerStarted","Data":"0e8007a3d5d659c479bce8f8b97b40d66ed88b758c68b6b642254c9ea672f8e3"} Nov 04 01:30:01.374301 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.374266 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n" event={"ID":"c1f011e7-dc3c-4b35-8c59-76bd4101237c","Type":"ContainerStarted","Data":"d2edcfe6fb154360019840ed3e79793029d753c3171b522715e54981010674a4"} Nov 04 01:30:01.389836 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.389805 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh" event={"ID":"48244ae6-4584-4874-b776-610df53ed021","Type":"ContainerStarted","Data":"eeb434ec0766fe9dd8842f5b1210a2fb191fe42398c506a26b455fd59dc7539d"} Nov 04 01:30:01.391474 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.391454 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n" event={"ID":"4a17cae9-e3d0-4a80-93ba-5602baba0d74","Type":"ContainerStarted","Data":"4c5a56664308ba80388a5a10bbf2b02706d4967ae43298abfab481021f66130f"} Nov 04 01:30:01.584368 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.581005 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7"] Nov 04 01:30:01.587915 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.587890 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.594124 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.594098 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-30b1a1-openshift-default-dockercfg-rbkbq\"" Nov 04 01:30:01.600996 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.600973 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7"] Nov 04 01:30:01.637727 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.637693 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-workload-socket\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.637855 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.637733 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-envoy\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.637922 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.637874 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-workload-certs\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.637970 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.637923 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s6lg\" (UniqueName: \"kubernetes.io/projected/207f3c0f-a068-4aee-a593-88e94bcc0dff-kube-api-access-9s6lg\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.638039 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.638022 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-token\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.638104 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.638086 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-credential-socket\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.638170 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.638137 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207f3c0f-a068-4aee-a593-88e94bcc0dff-istiod-ca-cert\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.638301 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.638189 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-data\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.638301 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.638274 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-podinfo\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.739643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.734714 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:01.774976 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.763424 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-workload-socket\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.774976 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.763486 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-envoy\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.774976 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.763576 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-workload-certs\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.774976 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.763618 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s6lg\" (UniqueName: \"kubernetes.io/projected/207f3c0f-a068-4aee-a593-88e94bcc0dff-kube-api-access-9s6lg\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.774976 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.763656 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-token\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.774976 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.763729 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-credential-socket\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.774976 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.763786 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207f3c0f-a068-4aee-a593-88e94bcc0dff-istiod-ca-cert\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.774976 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.763825 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-data\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.774976 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.763958 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-podinfo\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.804587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.801309 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-workload-socket\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.808303 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.806154 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207f3c0f-a068-4aee-a593-88e94bcc0dff-istiod-ca-cert\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.808303 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.806409 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-workload-certs\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.827680 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.816268 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-podinfo\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.827680 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.816551 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-data\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.961589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.948146 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-credential-socket\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:01.961589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:01.948933 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s6lg\" (UniqueName: \"kubernetes.io/projected/207f3c0f-a068-4aee-a593-88e94bcc0dff-kube-api-access-9s6lg\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:02.092443 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:02.082793 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-envoy\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:02.243675 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:02.182445 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-token\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:02.406734 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:30:02.338364 4852 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f011e7_dc3c_4b35_8c59_76bd4101237c.slice/crio-conmon-9b9e3aba7ba0bf6968bf82c82f57183dc8a43cd2bc9af4f5b1cc202a47d68ec1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f011e7_dc3c_4b35_8c59_76bd4101237c.slice/crio-9b9e3aba7ba0bf6968bf82c82f57183dc8a43cd2bc9af4f5b1cc202a47d68ec1.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:30:02.406734 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:02.404103 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:02.518457 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:02.518223 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:02.636105 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:02.636067 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7" event={"ID":"b2788c94-d82f-4ac6-94db-71ff7aeec55d","Type":"ContainerStarted","Data":"19512f8d179235d498590e913bcc3015bcd3c6f0982d1e6fbfa8985a32faef4e"} Nov 04 01:30:02.636368 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:02.636346 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7" event={"ID":"b2788c94-d82f-4ac6-94db-71ff7aeec55d","Type":"ContainerStarted","Data":"71643a6ead70b7441a07d5d97fccf286af57d24872c3ac3b9c0cdbcc55ecdea3"} Nov 04 01:30:02.641375 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:02.641345 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n" event={"ID":"c1f011e7-dc3c-4b35-8c59-76bd4101237c","Type":"ContainerStarted","Data":"9b9e3aba7ba0bf6968bf82c82f57183dc8a43cd2bc9af4f5b1cc202a47d68ec1"} Nov 04 01:30:02.659035 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:02.646448 4852 scope.go:122] "RemoveContainer" containerID="9b9e3aba7ba0bf6968bf82c82f57183dc8a43cd2bc9af4f5b1cc202a47d68ec1" Nov 04 01:30:02.659035 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:02.656968 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh" event={"ID":"48244ae6-4584-4874-b776-610df53ed021","Type":"ContainerStarted","Data":"de5453cd22188045c3d59a728dd7ef5e347e7471842966d9f4d51a3943701d0e"} Nov 04 01:30:02.659035 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:02.657546 4852 scope.go:122] "RemoveContainer" containerID="de5453cd22188045c3d59a728dd7ef5e347e7471842966d9f4d51a3943701d0e" Nov 04 01:30:02.815624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:02.813545 4852 generic.go:356] "Generic (PLEG): container finished" podID="dc68a724-a4bd-436b-92ff-4433c18960f2" containerID="59cf968fd9422e99ebb5299c62f87358e8bbcbc66a61b9f2d1b2bf3430cce5d8" exitCode=60 Nov 04 01:30:02.815624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:02.815011 4852 scope.go:122] "RemoveContainer" containerID="59cf968fd9422e99ebb5299c62f87358e8bbcbc66a61b9f2d1b2bf3430cce5d8" Nov 04 01:30:02.815624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:02.815516 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4" event={"ID":"dc68a724-a4bd-436b-92ff-4433c18960f2","Type":"ContainerDied","Data":"59cf968fd9422e99ebb5299c62f87358e8bbcbc66a61b9f2d1b2bf3430cce5d8"} Nov 04 01:30:02.819876 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:02.817507 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8" podStartSLOduration=2.817471199 podStartE2EDuration="2.817471199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:02.710179766 +0000 UTC m=+1884.825289887" watchObservedRunningTime="2026-11-04 01:30:02.817471199 +0000 UTC m=+1884.932581319" Nov 04 01:30:02.819876 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:02.818226 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5" podStartSLOduration=2.818216001 podStartE2EDuration="2.818216001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:02.808103446 +0000 UTC m=+1884.923213565" watchObservedRunningTime="2026-11-04 01:30:02.818216001 +0000 UTC m=+1884.933326120" Nov 04 01:30:02.916108 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:02.915936 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:02.946856 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:02.946810 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7" podStartSLOduration=2.946796164 podStartE2EDuration="2.946796164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:02.941949852 +0000 UTC m=+1885.057059971" watchObservedRunningTime="2026-11-04 01:30:02.946796164 +0000 UTC m=+1885.061906282" Nov 04 01:30:02.992127 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:02.991192 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n" podStartSLOduration=2.991179176 podStartE2EDuration="2.991179176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:02.99024652 +0000 UTC m=+1885.105356639" watchObservedRunningTime="2026-11-04 01:30:02.991179176 +0000 UTC m=+1885.106289317" Nov 04 01:30:03.108159 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.107993 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7"] Nov 04 01:30:03.151308 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:30:03.151253 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207f3c0f_a068_4aee_a593_88e94bcc0dff.slice/crio-f2f3baee6afcaaf47e891fa12882c13cc7b38cff0141be1f518b3278fbd5d87f WatchSource:0}: Error finding container f2f3baee6afcaaf47e891fa12882c13cc7b38cff0141be1f518b3278fbd5d87f: Status 404 returned error can't find the container with id f2f3baee6afcaaf47e891fa12882c13cc7b38cff0141be1f518b3278fbd5d87f Nov 04 01:30:03.155008 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.154963 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:03.155109 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.155060 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:03.155147 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.155104 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:03.328519 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.326914 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:03.845624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.845349 4852 generic.go:356] "Generic (PLEG): container finished" podID="03f61cb5-7acc-4d99-9b95-2c7492876e68" containerID="20ed07dd05f332c460ff7fe87093093bfc57b82c9bbe985fdd624b870845eb84" exitCode=60 Nov 04 01:30:03.845624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.845474 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5" event={"ID":"03f61cb5-7acc-4d99-9b95-2c7492876e68","Type":"ContainerDied","Data":"20ed07dd05f332c460ff7fe87093093bfc57b82c9bbe985fdd624b870845eb84"} Nov 04 01:30:03.847619 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.846339 4852 scope.go:122] "RemoveContainer" containerID="20ed07dd05f332c460ff7fe87093093bfc57b82c9bbe985fdd624b870845eb84" Nov 04 01:30:03.850317 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.850287 4852 generic.go:356] "Generic (PLEG): container finished" podID="c1f011e7-dc3c-4b35-8c59-76bd4101237c" containerID="9b9e3aba7ba0bf6968bf82c82f57183dc8a43cd2bc9af4f5b1cc202a47d68ec1" exitCode=60 Nov 04 01:30:03.850430 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.850412 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n" event={"ID":"c1f011e7-dc3c-4b35-8c59-76bd4101237c","Type":"ContainerDied","Data":"9b9e3aba7ba0bf6968bf82c82f57183dc8a43cd2bc9af4f5b1cc202a47d68ec1"} Nov 04 01:30:03.850512 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.850457 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n" event={"ID":"c1f011e7-dc3c-4b35-8c59-76bd4101237c","Type":"ContainerStarted","Data":"de6c1bc0a8a6708a25c08db9a83d0b31ec2bbb0c89f050d5d3e19a5b7c859ae8"} Nov 04 01:30:03.854766 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.853208 4852 generic.go:356] "Generic (PLEG): container finished" podID="48244ae6-4584-4874-b776-610df53ed021" containerID="de5453cd22188045c3d59a728dd7ef5e347e7471842966d9f4d51a3943701d0e" exitCode=60 Nov 04 01:30:03.854766 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.853251 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh" event={"ID":"48244ae6-4584-4874-b776-610df53ed021","Type":"ContainerDied","Data":"de5453cd22188045c3d59a728dd7ef5e347e7471842966d9f4d51a3943701d0e"} Nov 04 01:30:03.854766 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.853312 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh" event={"ID":"48244ae6-4584-4874-b776-610df53ed021","Type":"ContainerStarted","Data":"0381c01349bfde95fbe8577356db23383f3251dc58f95cca4586d7b1d47a7019"} Nov 04 01:30:03.855276 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.855251 4852 generic.go:356] "Generic (PLEG): container finished" podID="4a17cae9-e3d0-4a80-93ba-5602baba0d74" containerID="4e193dca17e7da9539455d0499c992567fd53c2b224d9c74df305494f6f1b0d7" exitCode=60 Nov 04 01:30:03.855376 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.855358 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n" event={"ID":"4a17cae9-e3d0-4a80-93ba-5602baba0d74","Type":"ContainerDied","Data":"4e193dca17e7da9539455d0499c992567fd53c2b224d9c74df305494f6f1b0d7"} Nov 04 01:30:03.857317 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.857260 4852 scope.go:122] "RemoveContainer" containerID="4e193dca17e7da9539455d0499c992567fd53c2b224d9c74df305494f6f1b0d7" Nov 04 01:30:03.858066 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.858038 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4" event={"ID":"dc68a724-a4bd-436b-92ff-4433c18960f2","Type":"ContainerStarted","Data":"e839742534fc89907e414ceeb7dcf074543cbec2da3c283a2e312e4836cb7e75"} Nov 04 01:30:03.871148 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.860764 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" event={"ID":"207f3c0f-a068-4aee-a593-88e94bcc0dff","Type":"ContainerStarted","Data":"28dc1bfb2ceae7411142f3726755faf4c03092336b4000df1e7d2683547abe2d"} Nov 04 01:30:03.871148 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.860795 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" event={"ID":"207f3c0f-a068-4aee-a593-88e94bcc0dff","Type":"ContainerStarted","Data":"f2f3baee6afcaaf47e891fa12882c13cc7b38cff0141be1f518b3278fbd5d87f"} Nov 04 01:30:03.871148 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.863107 4852 generic.go:356] "Generic (PLEG): container finished" podID="d9585268-ca61-4539-84d5-a986d3161a1d" containerID="1155478cf05de88d8b2ac980504b3c6d684690c5f7e595d62d26b5ca9719b044" exitCode=60 Nov 04 01:30:03.871148 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.863248 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9" event={"ID":"d9585268-ca61-4539-84d5-a986d3161a1d","Type":"ContainerDied","Data":"1155478cf05de88d8b2ac980504b3c6d684690c5f7e595d62d26b5ca9719b044"} Nov 04 01:30:03.871148 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.863581 4852 scope.go:122] "RemoveContainer" containerID="1155478cf05de88d8b2ac980504b3c6d684690c5f7e595d62d26b5ca9719b044" Nov 04 01:30:03.871148 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.866887 4852 generic.go:356] "Generic (PLEG): container finished" podID="cb73d5de-5e15-4453-80fc-e643b3dbd961" containerID="f6375d9a29125db5979dc0ec061ebb4f46942f1da765deab023bda042a50aaa3" exitCode=60 Nov 04 01:30:03.871148 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.866961 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8" event={"ID":"cb73d5de-5e15-4453-80fc-e643b3dbd961","Type":"ContainerDied","Data":"f6375d9a29125db5979dc0ec061ebb4f46942f1da765deab023bda042a50aaa3"} Nov 04 01:30:03.871148 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.867374 4852 scope.go:122] "RemoveContainer" containerID="f6375d9a29125db5979dc0ec061ebb4f46942f1da765deab023bda042a50aaa3" Nov 04 01:30:03.872351 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.872329 4852 generic.go:356] "Generic (PLEG): container finished" podID="b2788c94-d82f-4ac6-94db-71ff7aeec55d" containerID="19512f8d179235d498590e913bcc3015bcd3c6f0982d1e6fbfa8985a32faef4e" exitCode=60 Nov 04 01:30:03.872450 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.872372 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7" event={"ID":"b2788c94-d82f-4ac6-94db-71ff7aeec55d","Type":"ContainerDied","Data":"19512f8d179235d498590e913bcc3015bcd3c6f0982d1e6fbfa8985a32faef4e"} Nov 04 01:30:03.872704 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.872684 4852 scope.go:122] "RemoveContainer" containerID="19512f8d179235d498590e913bcc3015bcd3c6f0982d1e6fbfa8985a32faef4e" Nov 04 01:30:03.901690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.901639 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh" podStartSLOduration=3.9016214380000003 podStartE2EDuration="3.901621438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:03.900064282 +0000 UTC m=+1886.015174401" watchObservedRunningTime="2026-11-04 01:30:03.901621438 +0000 UTC m=+1886.016731550" Nov 04 01:30:03.958447 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.958371 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n" podStartSLOduration=3.958352739 podStartE2EDuration="3.958352739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:03.945057807 +0000 UTC m=+1886.060167925" watchObservedRunningTime="2026-11-04 01:30:03.958352739 +0000 UTC m=+1886.073462859" Nov 04 01:30:03.977826 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:03.977780 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4" podStartSLOduration=3.9777658000000002 podStartE2EDuration="3.9777658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:03.973429243 +0000 UTC m=+1886.088539361" watchObservedRunningTime="2026-11-04 01:30:03.9777658 +0000 UTC m=+1886.092875918" Nov 04 01:30:04.040330 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.040280 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" podStartSLOduration=3.04026648 podStartE2EDuration="3.04026648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:04.036496905 +0000 UTC m=+1886.151607014" watchObservedRunningTime="2026-11-04 01:30:04.04026648 +0000 UTC m=+1886.155376598" Nov 04 01:30:04.440259 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.440207 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:04.526941 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.526903 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:04.535157 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.535126 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:04.883099 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.883069 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9" event={"ID":"d9585268-ca61-4539-84d5-a986d3161a1d","Type":"ContainerStarted","Data":"5860a8a0fe16980d0bae1fbddfcf346499719a15e95a8a9d1637051d74fd3bb1"} Nov 04 01:30:04.885224 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.885195 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8" event={"ID":"cb73d5de-5e15-4453-80fc-e643b3dbd961","Type":"ContainerStarted","Data":"cd895186d6851e713d77868e14804c9e48f958bc5277a93cdd13d16f88a60066"} Nov 04 01:30:04.887764 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.887727 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7" event={"ID":"b2788c94-d82f-4ac6-94db-71ff7aeec55d","Type":"ContainerStarted","Data":"9af08a7de3f4c5d27e589526c4d4c05b0c21d97446f56c47af2050f28379e77f"} Nov 04 01:30:04.889895 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.889869 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5" event={"ID":"03f61cb5-7acc-4d99-9b95-2c7492876e68","Type":"ContainerStarted","Data":"04d27d6dd7b8a98f3c7683bba7f441548b9b72d80f79aefb73c0396d9a6d9711"} Nov 04 01:30:04.891779 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.891756 4852 generic.go:356] "Generic (PLEG): container finished" podID="c1f011e7-dc3c-4b35-8c59-76bd4101237c" containerID="de6c1bc0a8a6708a25c08db9a83d0b31ec2bbb0c89f050d5d3e19a5b7c859ae8" exitCode=60 Nov 04 01:30:04.891927 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.891828 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n" event={"ID":"c1f011e7-dc3c-4b35-8c59-76bd4101237c","Type":"ContainerDied","Data":"de6c1bc0a8a6708a25c08db9a83d0b31ec2bbb0c89f050d5d3e19a5b7c859ae8"} Nov 04 01:30:04.891927 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.891881 4852 scope.go:122] "RemoveContainer" containerID="9b9e3aba7ba0bf6968bf82c82f57183dc8a43cd2bc9af4f5b1cc202a47d68ec1" Nov 04 01:30:04.892040 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.892003 4852 scope.go:122] "RemoveContainer" containerID="de6c1bc0a8a6708a25c08db9a83d0b31ec2bbb0c89f050d5d3e19a5b7c859ae8" Nov 04 01:30:04.892366 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:30:04.892331 4852 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n_odh-ai-gateway-infra(c1f011e7-dc3c-4b35-8c59-76bd4101237c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n" podUID="c1f011e7-dc3c-4b35-8c59-76bd4101237c" Nov 04 01:30:04.894204 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.894179 4852 generic.go:356] "Generic (PLEG): container finished" podID="48244ae6-4584-4874-b776-610df53ed021" containerID="0381c01349bfde95fbe8577356db23383f3251dc58f95cca4586d7b1d47a7019" exitCode=60 Nov 04 01:30:04.894309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.894267 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh" event={"ID":"48244ae6-4584-4874-b776-610df53ed021","Type":"ContainerDied","Data":"0381c01349bfde95fbe8577356db23383f3251dc58f95cca4586d7b1d47a7019"} Nov 04 01:30:04.894488 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.894473 4852 scope.go:122] "RemoveContainer" containerID="0381c01349bfde95fbe8577356db23383f3251dc58f95cca4586d7b1d47a7019" Nov 04 01:30:04.894817 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:30:04.894794 4852 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh_odh-ai-gateway-infra(48244ae6-4584-4874-b776-610df53ed021)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh" podUID="48244ae6-4584-4874-b776-610df53ed021" Nov 04 01:30:04.896382 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.896357 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n" event={"ID":"4a17cae9-e3d0-4a80-93ba-5602baba0d74","Type":"ContainerStarted","Data":"c1085f6c08b7bdc3dee9cb0ec9a97d1a9686a4cd0cdb36517a386f226d88f6d9"} Nov 04 01:30:04.898765 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.898739 4852 generic.go:356] "Generic (PLEG): container finished" podID="dc68a724-a4bd-436b-92ff-4433c18960f2" containerID="e839742534fc89907e414ceeb7dcf074543cbec2da3c283a2e312e4836cb7e75" exitCode=60 Nov 04 01:30:04.899242 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.899217 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4" event={"ID":"dc68a724-a4bd-436b-92ff-4433c18960f2","Type":"ContainerDied","Data":"e839742534fc89907e414ceeb7dcf074543cbec2da3c283a2e312e4836cb7e75"} Nov 04 01:30:04.899455 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.899438 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:04.899546 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.899505 4852 scope.go:122] "RemoveContainer" containerID="e839742534fc89907e414ceeb7dcf074543cbec2da3c283a2e312e4836cb7e75" Nov 04 01:30:04.899858 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:30:04.899831 4852 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4_odh-ai-gateway-infra(dc68a724-a4bd-436b-92ff-4433c18960f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4" podUID="dc68a724-a4bd-436b-92ff-4433c18960f2" Nov 04 01:30:04.900732 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.900716 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:30:04.908168 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.908128 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9" podStartSLOduration=4.908116115 podStartE2EDuration="4.908116115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:04.902509826 +0000 UTC m=+1887.017619944" watchObservedRunningTime="2026-11-04 01:30:04.908116115 +0000 UTC m=+1887.023226233" Nov 04 01:30:04.908730 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.908708 4852 scope.go:122] "RemoveContainer" containerID="de5453cd22188045c3d59a728dd7ef5e347e7471842966d9f4d51a3943701d0e" Nov 04 01:30:04.926718 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:04.926454 4852 scope.go:122] "RemoveContainer" containerID="59cf968fd9422e99ebb5299c62f87358e8bbcbc66a61b9f2d1b2bf3430cce5d8" Nov 04 01:30:05.368716 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.367970 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:05.909729 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.909696 4852 generic.go:356] "Generic (PLEG): container finished" podID="cb73d5de-5e15-4453-80fc-e643b3dbd961" containerID="cd895186d6851e713d77868e14804c9e48f958bc5277a93cdd13d16f88a60066" exitCode=60 Nov 04 01:30:05.910192 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.909776 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8" event={"ID":"cb73d5de-5e15-4453-80fc-e643b3dbd961","Type":"ContainerDied","Data":"cd895186d6851e713d77868e14804c9e48f958bc5277a93cdd13d16f88a60066"} Nov 04 01:30:05.910192 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.909858 4852 scope.go:122] "RemoveContainer" containerID="f6375d9a29125db5979dc0ec061ebb4f46942f1da765deab023bda042a50aaa3" Nov 04 01:30:05.910317 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.910241 4852 scope.go:122] "RemoveContainer" containerID="cd895186d6851e713d77868e14804c9e48f958bc5277a93cdd13d16f88a60066" Nov 04 01:30:05.910733 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:30:05.910694 4852 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8_odh-ai-gateway-infra(cb73d5de-5e15-4453-80fc-e643b3dbd961)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8" podUID="cb73d5de-5e15-4453-80fc-e643b3dbd961" Nov 04 01:30:05.912026 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.912006 4852 generic.go:356] "Generic (PLEG): container finished" podID="b2788c94-d82f-4ac6-94db-71ff7aeec55d" containerID="9af08a7de3f4c5d27e589526c4d4c05b0c21d97446f56c47af2050f28379e77f" exitCode=60 Nov 04 01:30:05.912125 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.912102 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7" event={"ID":"b2788c94-d82f-4ac6-94db-71ff7aeec55d","Type":"ContainerDied","Data":"9af08a7de3f4c5d27e589526c4d4c05b0c21d97446f56c47af2050f28379e77f"} Nov 04 01:30:05.912382 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.912362 4852 scope.go:122] "RemoveContainer" containerID="9af08a7de3f4c5d27e589526c4d4c05b0c21d97446f56c47af2050f28379e77f" Nov 04 01:30:05.912793 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:30:05.912767 4852 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7_odh-ai-gateway-infra(b2788c94-d82f-4ac6-94db-71ff7aeec55d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7" podUID="b2788c94-d82f-4ac6-94db-71ff7aeec55d" Nov 04 01:30:05.914190 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.914170 4852 generic.go:356] "Generic (PLEG): container finished" podID="03f61cb5-7acc-4d99-9b95-2c7492876e68" containerID="04d27d6dd7b8a98f3c7683bba7f441548b9b72d80f79aefb73c0396d9a6d9711" exitCode=60 Nov 04 01:30:05.914283 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.914243 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5" event={"ID":"03f61cb5-7acc-4d99-9b95-2c7492876e68","Type":"ContainerDied","Data":"04d27d6dd7b8a98f3c7683bba7f441548b9b72d80f79aefb73c0396d9a6d9711"} Nov 04 01:30:05.914657 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.914639 4852 scope.go:122] "RemoveContainer" containerID="04d27d6dd7b8a98f3c7683bba7f441548b9b72d80f79aefb73c0396d9a6d9711" Nov 04 01:30:05.914991 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:30:05.914970 4852 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5_odh-ai-gateway-infra(03f61cb5-7acc-4d99-9b95-2c7492876e68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5" podUID="03f61cb5-7acc-4d99-9b95-2c7492876e68" Nov 04 01:30:05.916435 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.916406 4852 scope.go:122] "RemoveContainer" containerID="de6c1bc0a8a6708a25c08db9a83d0b31ec2bbb0c89f050d5d3e19a5b7c859ae8" Nov 04 01:30:05.916796 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:30:05.916770 4852 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n_odh-ai-gateway-infra(c1f011e7-dc3c-4b35-8c59-76bd4101237c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n" podUID="c1f011e7-dc3c-4b35-8c59-76bd4101237c" Nov 04 01:30:05.918152 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.918135 4852 scope.go:122] "RemoveContainer" containerID="0381c01349bfde95fbe8577356db23383f3251dc58f95cca4586d7b1d47a7019" Nov 04 01:30:05.918447 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:30:05.918428 4852 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh_odh-ai-gateway-infra(48244ae6-4584-4874-b776-610df53ed021)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh" podUID="48244ae6-4584-4874-b776-610df53ed021" Nov 04 01:30:05.919911 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.919893 4852 generic.go:356] "Generic (PLEG): container finished" podID="4a17cae9-e3d0-4a80-93ba-5602baba0d74" containerID="c1085f6c08b7bdc3dee9cb0ec9a97d1a9686a4cd0cdb36517a386f226d88f6d9" exitCode=60 Nov 04 01:30:05.920010 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.919969 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n" event={"ID":"4a17cae9-e3d0-4a80-93ba-5602baba0d74","Type":"ContainerDied","Data":"c1085f6c08b7bdc3dee9cb0ec9a97d1a9686a4cd0cdb36517a386f226d88f6d9"} Nov 04 01:30:05.920298 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.920284 4852 scope.go:122] "RemoveContainer" containerID="c1085f6c08b7bdc3dee9cb0ec9a97d1a9686a4cd0cdb36517a386f226d88f6d9" Nov 04 01:30:05.920636 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:30:05.920618 4852 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n_odh-ai-gateway-infra(4a17cae9-e3d0-4a80-93ba-5602baba0d74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n" podUID="4a17cae9-e3d0-4a80-93ba-5602baba0d74" Nov 04 01:30:05.921863 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.921842 4852 scope.go:122] "RemoveContainer" containerID="e839742534fc89907e414ceeb7dcf074543cbec2da3c283a2e312e4836cb7e75" Nov 04 01:30:05.922673 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:30:05.922129 4852 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4_odh-ai-gateway-infra(dc68a724-a4bd-436b-92ff-4433c18960f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4" podUID="dc68a724-a4bd-436b-92ff-4433c18960f2" Nov 04 01:30:05.925665 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.925645 4852 scope.go:122] "RemoveContainer" containerID="19512f8d179235d498590e913bcc3015bcd3c6f0982d1e6fbfa8985a32faef4e" Nov 04 01:30:05.931537 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.931500 4852 generic.go:356] "Generic (PLEG): container finished" podID="d9585268-ca61-4539-84d5-a986d3161a1d" containerID="5860a8a0fe16980d0bae1fbddfcf346499719a15e95a8a9d1637051d74fd3bb1" exitCode=60 Nov 04 01:30:05.931732 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.931703 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9" event={"ID":"d9585268-ca61-4539-84d5-a986d3161a1d","Type":"ContainerDied","Data":"5860a8a0fe16980d0bae1fbddfcf346499719a15e95a8a9d1637051d74fd3bb1"} Nov 04 01:30:05.932444 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.932426 4852 scope.go:122] "RemoveContainer" containerID="5860a8a0fe16980d0bae1fbddfcf346499719a15e95a8a9d1637051d74fd3bb1" Nov 04 01:30:05.932913 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:30:05.932883 4852 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9_odh-ai-gateway-infra(d9585268-ca61-4539-84d5-a986d3161a1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9" podUID="d9585268-ca61-4539-84d5-a986d3161a1d" Nov 04 01:30:05.946218 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.946194 4852 scope.go:122] "RemoveContainer" containerID="20ed07dd05f332c460ff7fe87093093bfc57b82c9bbe985fdd624b870845eb84" Nov 04 01:30:05.960517 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.960487 4852 scope.go:122] "RemoveContainer" containerID="4e193dca17e7da9539455d0499c992567fd53c2b224d9c74df305494f6f1b0d7" Nov 04 01:30:05.975754 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:05.975733 4852 scope.go:122] "RemoveContainer" containerID="1155478cf05de88d8b2ac980504b3c6d684690c5f7e595d62d26b5ca9719b044" Nov 04 01:30:06.570527 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:06.570494 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:06.935072 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:06.934962 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:07.223650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:07.220823 4852 scope.go:122] "RemoveContainer" containerID="5860a8a0fe16980d0bae1fbddfcf346499719a15e95a8a9d1637051d74fd3bb1" Nov 04 01:30:07.223650 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:30:07.221192 4852 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9_odh-ai-gateway-infra(d9585268-ca61-4539-84d5-a986d3161a1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9" podUID="d9585268-ca61-4539-84d5-a986d3161a1d" Nov 04 01:30:07.240590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:07.229014 4852 scope.go:122] "RemoveContainer" containerID="9af08a7de3f4c5d27e589526c4d4c05b0c21d97446f56c47af2050f28379e77f" Nov 04 01:30:07.240590 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:30:07.229423 4852 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7_odh-ai-gateway-infra(b2788c94-d82f-4ac6-94db-71ff7aeec55d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7" podUID="b2788c94-d82f-4ac6-94db-71ff7aeec55d" Nov 04 01:30:07.279637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:07.260655 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:07.570577 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:07.568455 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:08.232858 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:08.232820 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:09.767429 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:09.767397 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:09.943588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:09.938047 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:11.274923 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:11.274888 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:11.290142 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:11.290108 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8"] Nov 04 01:30:11.290334 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:11.290314 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" podUID="f5744bca-e67c-4284-b08e-1a3cc950ca77" containerName="maas-api" containerID="cri-o://50e44dfd0f5152526674b042e13e7b01766526e496437eaf6612e993a616956b" gracePeriod=30 Nov 04 01:30:11.455617 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:11.455085 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:11.455617 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:11.455106 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9"] Nov 04 01:30:11.626125 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:11.626039 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:11.650186 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:11.650146 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" Nov 04 01:30:11.693593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:11.685069 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8"] Nov 04 01:30:11.693593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:11.686345 4852 cpu_manager.go:414] "containerMap: removing container" podUID="f5744bca-e67c-4284-b08e-1a3cc950ca77" containerName="maas-api" Nov 04 01:30:11.693593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:11.686365 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5744bca-e67c-4284-b08e-1a3cc950ca77" containerName="maas-api" Nov 04 01:30:11.693593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:11.686544 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5744bca-e67c-4284-b08e-1a3cc950ca77" containerName="maas-api" Nov 04 01:30:11.693593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:11.691206 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:11.739621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:11.709946 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8"] Nov 04 01:30:11.963777 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:11.928827 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5744bca-e67c-4284-b08e-1a3cc950ca77-kube-api-access-zppgx\" (UniqueName: \"kubernetes.io/projected/f5744bca-e67c-4284-b08e-1a3cc950ca77-kube-api-access-zppgx\") pod \"f5744bca-e67c-4284-b08e-1a3cc950ca77\" (UID: \"f5744bca-e67c-4284-b08e-1a3cc950ca77\") " Nov 04 01:30:11.973588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:11.967953 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5744bca-e67c-4284-b08e-1a3cc950ca77-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5744bca-e67c-4284-b08e-1a3cc950ca77-openshift-service-ca\") pod \"f5744bca-e67c-4284-b08e-1a3cc950ca77\" (UID: \"f5744bca-e67c-4284-b08e-1a3cc950ca77\") " Nov 04 01:30:11.984043 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:11.984019 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5744bca-e67c-4284-b08e-1a3cc950ca77-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5744bca-e67c-4284-b08e-1a3cc950ca77-maas-api-tls\") pod \"f5744bca-e67c-4284-b08e-1a3cc950ca77\" (UID: \"f5744bca-e67c-4284-b08e-1a3cc950ca77\") " Nov 04 01:30:11.984736 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:11.984714 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5744bca-e67c-4284-b08e-1a3cc950ca77-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5744bca-e67c-4284-b08e-1a3cc950ca77-metrics-certs\") pod \"f5744bca-e67c-4284-b08e-1a3cc950ca77\" (UID: \"f5744bca-e67c-4284-b08e-1a3cc950ca77\") " Nov 04 01:30:12.011667 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:11.983920 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5744bca-e67c-4284-b08e-1a3cc950ca77-openshift-service-ca" pod "f5744bca-e67c-4284-b08e-1a3cc950ca77" (UID: "f5744bca-e67c-4284-b08e-1a3cc950ca77"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:12.025052 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.023917 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtdwf\" (UniqueName: \"kubernetes.io/projected/107a1fd6-feb8-4615-91ef-980f9af78d5f-kube-api-access-mtdwf\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.025052 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.024076 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.025052 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.024173 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.025052 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.024338 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.025052 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.024406 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.025052 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.024469 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.025052 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.024515 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.025052 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.024538 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107a1fd6-feb8-4615-91ef-980f9af78d5f-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.025052 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.024752 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.025052 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.024872 4852 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5744bca-e67c-4284-b08e-1a3cc950ca77-openshift-service-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:12.155705 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.126163 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.155705 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.126346 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.155705 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.126415 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.155705 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.126482 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.155705 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.126541 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.155705 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.126583 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107a1fd6-feb8-4615-91ef-980f9af78d5f-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.155705 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.126772 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.155705 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.126810 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtdwf\" (UniqueName: \"kubernetes.io/projected/107a1fd6-feb8-4615-91ef-980f9af78d5f-kube-api-access-mtdwf\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.155705 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.126962 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.155705 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.127468 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.155705 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.130576 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107a1fd6-feb8-4615-91ef-980f9af78d5f-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.155705 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.130850 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.158946 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.157730 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.158946 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.158419 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.164932 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.162505 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.164932 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.162810 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.188723 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.187915 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtdwf\" (UniqueName: \"kubernetes.io/projected/107a1fd6-feb8-4615-91ef-980f9af78d5f-kube-api-access-mtdwf\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.245674 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.243443 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9" Nov 04 01:30:12.328221 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.328187 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5744bca-e67c-4284-b08e-1a3cc950ca77-maas-api-tls" pod "f5744bca-e67c-4284-b08e-1a3cc950ca77" (UID: "f5744bca-e67c-4284-b08e-1a3cc950ca77"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:30:12.330973 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.330941 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.333604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.333584 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9585268-ca61-4539-84d5-a986d3161a1d-kube-api-access-qdpzp\" (UniqueName: \"kubernetes.io/projected/d9585268-ca61-4539-84d5-a986d3161a1d-kube-api-access-qdpzp\") pod \"d9585268-ca61-4539-84d5-a986d3161a1d\" (UID: \"d9585268-ca61-4539-84d5-a986d3161a1d\") " Nov 04 01:30:12.334090 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.334076 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9585268-ca61-4539-84d5-a986d3161a1d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9585268-ca61-4539-84d5-a986d3161a1d-maas-api-service-ca\") pod \"d9585268-ca61-4539-84d5-a986d3161a1d\" (UID: \"d9585268-ca61-4539-84d5-a986d3161a1d\") " Nov 04 01:30:12.335092 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.335063 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9585268-ca61-4539-84d5-a986d3161a1d-maas-api-service-ca" pod "d9585268-ca61-4539-84d5-a986d3161a1d" (UID: "d9585268-ca61-4539-84d5-a986d3161a1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:12.335298 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.335279 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5744bca-e67c-4284-b08e-1a3cc950ca77-maas-api-tls\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:12.335369 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.335304 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9585268-ca61-4539-84d5-a986d3161a1d-maas-api-service-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:12.337678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.337343 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5744bca-e67c-4284-b08e-1a3cc950ca77-kube-api-access-zppgx" pod "f5744bca-e67c-4284-b08e-1a3cc950ca77" (UID: "f5744bca-e67c-4284-b08e-1a3cc950ca77"). InnerVolumeSpecName "kube-api-access-zppgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:12.337678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.337406 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5744bca-e67c-4284-b08e-1a3cc950ca77-metrics-certs" pod "f5744bca-e67c-4284-b08e-1a3cc950ca77" (UID: "f5744bca-e67c-4284-b08e-1a3cc950ca77"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:30:12.351711 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.351682 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9585268-ca61-4539-84d5-a986d3161a1d-kube-api-access-qdpzp" pod "d9585268-ca61-4539-84d5-a986d3161a1d" (UID: "d9585268-ca61-4539-84d5-a986d3161a1d"). InnerVolumeSpecName "kube-api-access-qdpzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:12.391794 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.391764 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:12.404585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.403633 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9" Nov 04 01:30:12.404585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.403684 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9" event={"ID":"d9585268-ca61-4539-84d5-a986d3161a1d","Type":"ContainerDied","Data":"efa0d13a7bd175f4f1ffe9a895c24be37da02aab9189106a49c26128ca06096e"} Nov 04 01:30:12.404585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.403749 4852 scope.go:122] "RemoveContainer" containerID="5860a8a0fe16980d0bae1fbddfcf346499719a15e95a8a9d1637051d74fd3bb1" Nov 04 01:30:12.411896 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.411865 4852 generic.go:356] "Generic (PLEG): container finished" podID="f5744bca-e67c-4284-b08e-1a3cc950ca77" containerID="50e44dfd0f5152526674b042e13e7b01766526e496437eaf6612e993a616956b" exitCode=0 Nov 04 01:30:12.412024 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.411911 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" Nov 04 01:30:12.412024 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.411932 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" event={"ID":"f5744bca-e67c-4284-b08e-1a3cc950ca77","Type":"ContainerDied","Data":"50e44dfd0f5152526674b042e13e7b01766526e496437eaf6612e993a616956b"} Nov 04 01:30:12.412024 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.411969 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8" event={"ID":"f5744bca-e67c-4284-b08e-1a3cc950ca77","Type":"ContainerDied","Data":"6d7ce0b886111d41798f2d59ce2b5f82257b88b4ed337de053c331a5e787bc9c"} Nov 04 01:30:12.436690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.436660 4852 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5744bca-e67c-4284-b08e-1a3cc950ca77-metrics-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:12.436690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.436687 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdpzp\" (UniqueName: \"kubernetes.io/projected/d9585268-ca61-4539-84d5-a986d3161a1d-kube-api-access-qdpzp\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:12.436690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.436699 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zppgx\" (UniqueName: \"kubernetes.io/projected/f5744bca-e67c-4284-b08e-1a3cc950ca77-kube-api-access-zppgx\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:12.446795 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.446760 4852 scope.go:122] "RemoveContainer" containerID="50e44dfd0f5152526674b042e13e7b01766526e496437eaf6612e993a616956b" Nov 04 01:30:12.459715 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.459623 4852 scope.go:122] "RemoveContainer" containerID="50e44dfd0f5152526674b042e13e7b01766526e496437eaf6612e993a616956b" Nov 04 01:30:12.460078 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:30:12.460053 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e44dfd0f5152526674b042e13e7b01766526e496437eaf6612e993a616956b\": container with ID starting with 50e44dfd0f5152526674b042e13e7b01766526e496437eaf6612e993a616956b not found: ID does not exist" containerID="50e44dfd0f5152526674b042e13e7b01766526e496437eaf6612e993a616956b" Nov 04 01:30:12.460165 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.460097 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e44dfd0f5152526674b042e13e7b01766526e496437eaf6612e993a616956b"} err="failed to get container status \"50e44dfd0f5152526674b042e13e7b01766526e496437eaf6612e993a616956b\": rpc error: code = NotFound desc = could not find container \"50e44dfd0f5152526674b042e13e7b01766526e496437eaf6612e993a616956b\": container with ID starting with 50e44dfd0f5152526674b042e13e7b01766526e496437eaf6612e993a616956b not found: ID does not exist" Nov 04 01:30:12.469520 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.469480 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8"] Nov 04 01:30:12.476379 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.476355 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee876a-55675db58f-f2xf8"] Nov 04 01:30:12.488913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.488888 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9"] Nov 04 01:30:12.623185 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.622592 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:12.653172 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.653137 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ee876a-29895930-tlxm9"] Nov 04 01:30:12.727555 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.727516 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8"] Nov 04 01:30:12.743704 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.743656 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:12.745870 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.745055 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:12.745870 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:12.745138 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:13.424087 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:13.424052 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" event={"ID":"107a1fd6-feb8-4615-91ef-980f9af78d5f","Type":"ContainerStarted","Data":"da3bc239519edadea7f880ceaaef53b2fa50c586d65aff4fcada07143be32366"} Nov 04 01:30:13.424530 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:13.424107 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" event={"ID":"107a1fd6-feb8-4615-91ef-980f9af78d5f","Type":"ContainerStarted","Data":"fd39a7e86826d2f223c18ab7a2b5a9c80d433d3b0ab3d969853803231bb26488"} Nov 04 01:30:13.458943 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:13.458868 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" podStartSLOduration=12.458835212 podStartE2EDuration="12.458835212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:13.455632476 +0000 UTC m=+1895.570742600" watchObservedRunningTime="2026-11-04 01:30:13.458835212 +0000 UTC m=+1895.573945332" Nov 04 01:30:13.623720 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:13.622963 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:13.997936 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:13.997905 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9585268-ca61-4539-84d5-a986d3161a1d" path="/var/lib/kubelet/pods/d9585268-ca61-4539-84d5-a986d3161a1d/volumes" Nov 04 01:30:13.998481 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:13.998459 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5744bca-e67c-4284-b08e-1a3cc950ca77" path="/var/lib/kubelet/pods/f5744bca-e67c-4284-b08e-1a3cc950ca77/volumes" Nov 04 01:30:14.230603 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:14.222772 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:14.398330 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:14.398287 4852 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:30:14.398638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:14.398365 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" podUID="107a1fd6-feb8-4615-91ef-980f9af78d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 04 01:30:14.399336 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:14.399311 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:14.852544 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:14.852511 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:15.154860 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:15.154784 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:15.394093 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:15.394061 4852 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:30:15.394260 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:15.394113 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" podUID="107a1fd6-feb8-4615-91ef-980f9af78d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 04 01:30:16.052591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:16.050960 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:16.394197 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:16.394100 4852 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:30:16.394197 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:16.394149 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" podUID="107a1fd6-feb8-4615-91ef-980f9af78d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 04 01:30:16.883140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:16.883104 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:16.990627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:16.987625 4852 scope.go:122] "RemoveContainer" containerID="de6c1bc0a8a6708a25c08db9a83d0b31ec2bbb0c89f050d5d3e19a5b7c859ae8" Nov 04 01:30:16.990627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:16.988814 4852 scope.go:122] "RemoveContainer" containerID="0381c01349bfde95fbe8577356db23383f3251dc58f95cca4586d7b1d47a7019" Nov 04 01:30:16.990627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:16.989601 4852 scope.go:122] "RemoveContainer" containerID="c1085f6c08b7bdc3dee9cb0ec9a97d1a9686a4cd0cdb36517a386f226d88f6d9" Nov 04 01:30:17.000882 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:17.000856 4852 scope.go:122] "RemoveContainer" containerID="e839742534fc89907e414ceeb7dcf074543cbec2da3c283a2e312e4836cb7e75" Nov 04 01:30:17.003182 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:17.003165 4852 scope.go:122] "RemoveContainer" containerID="04d27d6dd7b8a98f3c7683bba7f441548b9b72d80f79aefb73c0396d9a6d9711" Nov 04 01:30:17.393808 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:17.393776 4852 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:30:17.394173 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:17.393825 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" podUID="107a1fd6-feb8-4615-91ef-980f9af78d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 04 01:30:17.957484 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:17.957399 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:18.080090 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:18.080057 4852 scope.go:122] "RemoveContainer" containerID="cd895186d6851e713d77868e14804c9e48f958bc5277a93cdd13d16f88a60066" Nov 04 01:30:18.422607 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:18.418426 4852 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:30:18.422607 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:18.418495 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" podUID="107a1fd6-feb8-4615-91ef-980f9af78d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 04 01:30:18.517635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:18.517512 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n" event={"ID":"4a17cae9-e3d0-4a80-93ba-5602baba0d74","Type":"ContainerStarted","Data":"daa7ef8d4ba63ac94887500a9a7b50d3ed4851d693ab0574a516387571fa2f03"} Nov 04 01:30:18.519671 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:18.519646 4852 generic.go:356] "Generic (PLEG): container finished" podID="dc68a724-a4bd-436b-92ff-4433c18960f2" containerID="27775c8ce79fed938f1194305a24696ad9b6a7f8d557472fa1b1a6d1d5dde93a" exitCode=60 Nov 04 01:30:18.519828 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:18.519728 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4" event={"ID":"dc68a724-a4bd-436b-92ff-4433c18960f2","Type":"ContainerDied","Data":"27775c8ce79fed938f1194305a24696ad9b6a7f8d557472fa1b1a6d1d5dde93a"} Nov 04 01:30:18.519828 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:18.519789 4852 scope.go:122] "RemoveContainer" containerID="e839742534fc89907e414ceeb7dcf074543cbec2da3c283a2e312e4836cb7e75" Nov 04 01:30:18.520262 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:18.520175 4852 scope.go:122] "RemoveContainer" containerID="27775c8ce79fed938f1194305a24696ad9b6a7f8d557472fa1b1a6d1d5dde93a" Nov 04 01:30:18.520544 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:30:18.520479 4852 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4_odh-ai-gateway-infra(dc68a724-a4bd-436b-92ff-4433c18960f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4" podUID="dc68a724-a4bd-436b-92ff-4433c18960f2" Nov 04 01:30:18.522444 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:18.522420 4852 generic.go:356] "Generic (PLEG): container finished" podID="03f61cb5-7acc-4d99-9b95-2c7492876e68" containerID="0543c49d7933394b6322e8e618b589fb842fb08bd3ecee9cae915b7bab3f1a05" exitCode=60 Nov 04 01:30:18.522538 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:18.522502 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5" event={"ID":"03f61cb5-7acc-4d99-9b95-2c7492876e68","Type":"ContainerDied","Data":"0543c49d7933394b6322e8e618b589fb842fb08bd3ecee9cae915b7bab3f1a05"} Nov 04 01:30:18.522937 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:18.522918 4852 scope.go:122] "RemoveContainer" containerID="0543c49d7933394b6322e8e618b589fb842fb08bd3ecee9cae915b7bab3f1a05" Nov 04 01:30:18.523402 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:30:18.523256 4852 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5_odh-ai-gateway-infra(03f61cb5-7acc-4d99-9b95-2c7492876e68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5" podUID="03f61cb5-7acc-4d99-9b95-2c7492876e68" Nov 04 01:30:18.524224 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:18.524205 4852 generic.go:356] "Generic (PLEG): container finished" podID="c1f011e7-dc3c-4b35-8c59-76bd4101237c" containerID="def465d755873db323adb64dc03e7c173db265996e31868fe54b66e565efd8e1" exitCode=60 Nov 04 01:30:18.524291 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:18.524272 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n" event={"ID":"c1f011e7-dc3c-4b35-8c59-76bd4101237c","Type":"ContainerDied","Data":"def465d755873db323adb64dc03e7c173db265996e31868fe54b66e565efd8e1"} Nov 04 01:30:18.524550 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:18.524533 4852 scope.go:122] "RemoveContainer" containerID="def465d755873db323adb64dc03e7c173db265996e31868fe54b66e565efd8e1" Nov 04 01:30:18.525356 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:30:18.525334 4852 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n_odh-ai-gateway-infra(c1f011e7-dc3c-4b35-8c59-76bd4101237c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n" podUID="c1f011e7-dc3c-4b35-8c59-76bd4101237c" Nov 04 01:30:18.526979 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:18.526960 4852 generic.go:356] "Generic (PLEG): container finished" podID="48244ae6-4584-4874-b776-610df53ed021" containerID="cfe6f7fb10df0aab58e506611a405a285ca3978aa680b16e88d378c89cfe8bf1" exitCode=60 Nov 04 01:30:18.527047 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:18.526990 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh" event={"ID":"48244ae6-4584-4874-b776-610df53ed021","Type":"ContainerDied","Data":"cfe6f7fb10df0aab58e506611a405a285ca3978aa680b16e88d378c89cfe8bf1"} Nov 04 01:30:18.527246 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:18.527230 4852 scope.go:122] "RemoveContainer" containerID="cfe6f7fb10df0aab58e506611a405a285ca3978aa680b16e88d378c89cfe8bf1" Nov 04 01:30:18.527442 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:30:18.527426 4852 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh_odh-ai-gateway-infra(48244ae6-4584-4874-b776-610df53ed021)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh" podUID="48244ae6-4584-4874-b776-610df53ed021" Nov 04 01:30:18.597321 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:30:18.597290 4852 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48244ae6_4584_4874_b776_610df53ed021.slice/crio-cfe6f7fb10df0aab58e506611a405a285ca3978aa680b16e88d378c89cfe8bf1.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:30:18.653871 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:18.653842 4852 scope.go:122] "RemoveContainer" containerID="04d27d6dd7b8a98f3c7683bba7f441548b9b72d80f79aefb73c0396d9a6d9711" Nov 04 01:30:18.694350 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:18.694318 4852 scope.go:122] "RemoveContainer" containerID="de6c1bc0a8a6708a25c08db9a83d0b31ec2bbb0c89f050d5d3e19a5b7c859ae8" Nov 04 01:30:18.711151 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:18.711118 4852 scope.go:122] "RemoveContainer" containerID="0381c01349bfde95fbe8577356db23383f3251dc58f95cca4586d7b1d47a7019" Nov 04 01:30:18.900641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:18.894994 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:18.989814 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:18.989209 4852 scope.go:122] "RemoveContainer" containerID="9af08a7de3f4c5d27e589526c4d4c05b0c21d97446f56c47af2050f28379e77f" Nov 04 01:30:19.409961 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:19.401640 4852 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:30:19.409961 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:19.401710 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" podUID="107a1fd6-feb8-4615-91ef-980f9af78d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 04 01:30:19.731642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:19.730890 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n"] Nov 04 01:30:19.773018 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:19.772968 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8" event={"ID":"cb73d5de-5e15-4453-80fc-e643b3dbd961","Type":"ContainerStarted","Data":"b8476269693139c9ff75cc64facf9c8947cdbf71341d3a28ae724b710b7ec94a"} Nov 04 01:30:19.796975 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:19.796933 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7" event={"ID":"b2788c94-d82f-4ac6-94db-71ff7aeec55d","Type":"ContainerStarted","Data":"8658bf81b3e3c0b67eb030fa16890a473bd4e652a78a7650661df7abb961db84"} Nov 04 01:30:19.856037 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:19.856003 4852 generic.go:356] "Generic (PLEG): container finished" podID="4a17cae9-e3d0-4a80-93ba-5602baba0d74" containerID="daa7ef8d4ba63ac94887500a9a7b50d3ed4851d693ab0574a516387571fa2f03" exitCode=60 Nov 04 01:30:19.856847 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:19.856805 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n" event={"ID":"4a17cae9-e3d0-4a80-93ba-5602baba0d74","Type":"ContainerDied","Data":"daa7ef8d4ba63ac94887500a9a7b50d3ed4851d693ab0574a516387571fa2f03"} Nov 04 01:30:19.859060 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:19.859042 4852 scope.go:122] "RemoveContainer" containerID="c1085f6c08b7bdc3dee9cb0ec9a97d1a9686a4cd0cdb36517a386f226d88f6d9" Nov 04 01:30:19.862054 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:19.862036 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh"] Nov 04 01:30:19.889073 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:19.877421 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n"] Nov 04 01:30:19.926594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:19.924302 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4"] Nov 04 01:30:19.931784 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:19.931760 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5"] Nov 04 01:30:20.240429 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:20.237137 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:20.454740 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:20.454453 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:20.456205 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:20.456184 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:20.531118 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:20.530327 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:30:20.917586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:20.915780 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8"] Nov 04 01:30:20.927971 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:20.927855 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7"] Nov 04 01:30:20.953601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:20.943749 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:21.479089 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:21.461594 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:21.642823 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:21.642784 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n" Nov 04 01:30:21.683028 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:21.652910 4852 generic.go:356] "Generic (PLEG): container finished" podID="cb73d5de-5e15-4453-80fc-e643b3dbd961" containerID="b8476269693139c9ff75cc64facf9c8947cdbf71341d3a28ae724b710b7ec94a" exitCode=60 Nov 04 01:30:21.683028 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:21.653009 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8" event={"ID":"cb73d5de-5e15-4453-80fc-e643b3dbd961","Type":"ContainerDied","Data":"b8476269693139c9ff75cc64facf9c8947cdbf71341d3a28ae724b710b7ec94a"} Nov 04 01:30:21.683028 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:21.653068 4852 scope.go:122] "RemoveContainer" containerID="cd895186d6851e713d77868e14804c9e48f958bc5277a93cdd13d16f88a60066" Nov 04 01:30:21.683028 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:21.676021 4852 generic.go:356] "Generic (PLEG): container finished" podID="b2788c94-d82f-4ac6-94db-71ff7aeec55d" containerID="8658bf81b3e3c0b67eb030fa16890a473bd4e652a78a7650661df7abb961db84" exitCode=60 Nov 04 01:30:21.683028 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:21.678043 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7" event={"ID":"b2788c94-d82f-4ac6-94db-71ff7aeec55d","Type":"ContainerDied","Data":"8658bf81b3e3c0b67eb030fa16890a473bd4e652a78a7650661df7abb961db84"} Nov 04 01:30:21.689053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:21.689016 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:21.923622 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:21.923590 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1f011e7-dc3c-4b35-8c59-76bd4101237c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f011e7-dc3c-4b35-8c59-76bd4101237c-maas-api-service-ca\") pod \"c1f011e7-dc3c-4b35-8c59-76bd4101237c\" (UID: \"c1f011e7-dc3c-4b35-8c59-76bd4101237c\") " Nov 04 01:30:21.924251 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:21.924229 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f011e7-dc3c-4b35-8c59-76bd4101237c-kube-api-access-vcd5h\" (UniqueName: \"kubernetes.io/projected/c1f011e7-dc3c-4b35-8c59-76bd4101237c-kube-api-access-vcd5h\") pod \"c1f011e7-dc3c-4b35-8c59-76bd4101237c\" (UID: \"c1f011e7-dc3c-4b35-8c59-76bd4101237c\") " Nov 04 01:30:21.931965 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:21.931936 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f011e7-dc3c-4b35-8c59-76bd4101237c-maas-api-service-ca" pod "c1f011e7-dc3c-4b35-8c59-76bd4101237c" (UID: "c1f011e7-dc3c-4b35-8c59-76bd4101237c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:21.958586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:21.951613 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f011e7-dc3c-4b35-8c59-76bd4101237c-maas-api-service-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:22.128203 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.128170 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:22.376604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.362688 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 04 01:30:22.376604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.363127 4852 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:30:22.376604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.363173 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" podUID="3a4d4cee-bd81-4a99-8d45-c636a02c64cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:30:22.380606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.378415 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:30:22.380606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.379591 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:30:22.415811 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.415783 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4" Nov 04 01:30:22.469659 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.469629 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:22.679654 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.645984 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc68a724-a4bd-436b-92ff-4433c18960f2-kube-api-access-59c7b\" (UniqueName: \"kubernetes.io/projected/dc68a724-a4bd-436b-92ff-4433c18960f2-kube-api-access-59c7b\") pod \"dc68a724-a4bd-436b-92ff-4433c18960f2\" (UID: \"dc68a724-a4bd-436b-92ff-4433c18960f2\") " Nov 04 01:30:22.712591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.680358 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc68a724-a4bd-436b-92ff-4433c18960f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc68a724-a4bd-436b-92ff-4433c18960f2-maas-api-service-ca\") pod \"dc68a724-a4bd-436b-92ff-4433c18960f2\" (UID: \"dc68a724-a4bd-436b-92ff-4433c18960f2\") " Nov 04 01:30:22.736604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.693086 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc68a724-a4bd-436b-92ff-4433c18960f2-maas-api-service-ca" pod "dc68a724-a4bd-436b-92ff-4433c18960f2" (UID: "dc68a724-a4bd-436b-92ff-4433c18960f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:22.736604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.693155 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 04 01:30:22.872807 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.821521 4852 scope.go:122] "RemoveContainer" containerID="9af08a7de3f4c5d27e589526c4d4c05b0c21d97446f56c47af2050f28379e77f" Nov 04 01:30:22.872807 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.843898 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc68a724-a4bd-436b-92ff-4433c18960f2-maas-api-service-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:22.922199 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.922165 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": context deadline exceeded" Nov 04 01:30:22.925661 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.925324 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f011e7-dc3c-4b35-8c59-76bd4101237c-kube-api-access-vcd5h" pod "c1f011e7-dc3c-4b35-8c59-76bd4101237c" (UID: "c1f011e7-dc3c-4b35-8c59-76bd4101237c"). InnerVolumeSpecName "kube-api-access-vcd5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:22.931698 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.931631 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n" Nov 04 01:30:22.936607 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.934736 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc68a724-a4bd-436b-92ff-4433c18960f2-kube-api-access-59c7b" pod "dc68a724-a4bd-436b-92ff-4433c18960f2" (UID: "dc68a724-a4bd-436b-92ff-4433c18960f2"). InnerVolumeSpecName "kube-api-access-59c7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:22.945725 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.945690 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcd5h\" (UniqueName: \"kubernetes.io/projected/c1f011e7-dc3c-4b35-8c59-76bd4101237c-kube-api-access-vcd5h\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:22.945858 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.945842 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59c7b\" (UniqueName: \"kubernetes.io/projected/dc68a724-a4bd-436b-92ff-4433c18960f2-kube-api-access-59c7b\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:22.949874 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.949855 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl"] Nov 04 01:30:22.950361 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.950339 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" podUID="3a4d4cee-bd81-4a99-8d45-c636a02c64cf" containerName="istio-proxy" containerID="cri-o://435f0eb11db1abdba93331fd4fcda004baef191174a6fd5edfc3d881c235b2c4" gracePeriod=30 Nov 04 01:30:22.951947 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:22.951931 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5" Nov 04 01:30:23.005992 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.005912 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d"] Nov 04 01:30:23.006208 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.006176 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" podUID="fbc69a29-d23c-4da4-a6af-be04977ee457" containerName="istio-proxy" containerID="cri-o://3fd939b9ff86004e1acc4c88457314d2ca6faa02cbc7e28c2b59eacfe8bb2613" gracePeriod=30 Nov 04 01:30:23.011181 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.011051 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5" event={"ID":"03f61cb5-7acc-4d99-9b95-2c7492876e68","Type":"ContainerDied","Data":"0e8007a3d5d659c479bce8f8b97b40d66ed88b758c68b6b642254c9ea672f8e3"} Nov 04 01:30:23.011181 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.011120 4852 scope.go:122] "RemoveContainer" containerID="0543c49d7933394b6322e8e618b589fb842fb08bd3ecee9cae915b7bab3f1a05" Nov 04 01:30:23.011181 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.011067 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5" Nov 04 01:30:23.013212 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.012758 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp"] Nov 04 01:30:23.013212 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.012946 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" podUID="ad7e7c06-6bfc-4fee-a35a-43b73e501458" containerName="istio-proxy" containerID="cri-o://a8194384632e59c0650c08d6e51a91c98c567f33611eb537e3874f6bc5edba97" gracePeriod=30 Nov 04 01:30:23.015520 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.015441 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh" Nov 04 01:30:23.016642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.016283 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n" event={"ID":"c1f011e7-dc3c-4b35-8c59-76bd4101237c","Type":"ContainerDied","Data":"d2edcfe6fb154360019840ed3e79793029d753c3171b522715e54981010674a4"} Nov 04 01:30:23.016642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.016380 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n" Nov 04 01:30:23.019036 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.018790 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n" event={"ID":"4a17cae9-e3d0-4a80-93ba-5602baba0d74","Type":"ContainerDied","Data":"4c5a56664308ba80388a5a10bbf2b02706d4967ae43298abfab481021f66130f"} Nov 04 01:30:23.019036 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.018889 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n" Nov 04 01:30:23.023714 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.022254 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4" Nov 04 01:30:23.025700 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.025678 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4" event={"ID":"dc68a724-a4bd-436b-92ff-4433c18960f2","Type":"ContainerDied","Data":"65d9011f22f6b90bc71ced6f053dfa76f19bb7573efac81d98868126c92951b9"} Nov 04 01:30:23.048266 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.046548 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a17cae9-e3d0-4a80-93ba-5602baba0d74-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a17cae9-e3d0-4a80-93ba-5602baba0d74-maas-api-service-ca\") pod \"4a17cae9-e3d0-4a80-93ba-5602baba0d74\" (UID: \"4a17cae9-e3d0-4a80-93ba-5602baba0d74\") " Nov 04 01:30:23.048266 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.046724 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03f61cb5-7acc-4d99-9b95-2c7492876e68-kube-api-access-sgbvv\" (UniqueName: \"kubernetes.io/projected/03f61cb5-7acc-4d99-9b95-2c7492876e68-kube-api-access-sgbvv\") pod \"03f61cb5-7acc-4d99-9b95-2c7492876e68\" (UID: \"03f61cb5-7acc-4d99-9b95-2c7492876e68\") " Nov 04 01:30:23.048266 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.046816 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03f61cb5-7acc-4d99-9b95-2c7492876e68-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f61cb5-7acc-4d99-9b95-2c7492876e68-maas-api-service-ca\") pod \"03f61cb5-7acc-4d99-9b95-2c7492876e68\" (UID: \"03f61cb5-7acc-4d99-9b95-2c7492876e68\") " Nov 04 01:30:23.048266 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.046929 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a17cae9-e3d0-4a80-93ba-5602baba0d74-maas-api-service-ca" pod "4a17cae9-e3d0-4a80-93ba-5602baba0d74" (UID: "4a17cae9-e3d0-4a80-93ba-5602baba0d74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:23.048266 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.047083 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a17cae9-e3d0-4a80-93ba-5602baba0d74-kube-api-access-drdsq\" (UniqueName: \"kubernetes.io/projected/4a17cae9-e3d0-4a80-93ba-5602baba0d74-kube-api-access-drdsq\") pod \"4a17cae9-e3d0-4a80-93ba-5602baba0d74\" (UID: \"4a17cae9-e3d0-4a80-93ba-5602baba0d74\") " Nov 04 01:30:23.048266 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.047591 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f61cb5-7acc-4d99-9b95-2c7492876e68-maas-api-service-ca" pod "03f61cb5-7acc-4d99-9b95-2c7492876e68" (UID: "03f61cb5-7acc-4d99-9b95-2c7492876e68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:23.048266 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.048034 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a17cae9-e3d0-4a80-93ba-5602baba0d74-maas-api-service-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:23.048266 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.048054 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f61cb5-7acc-4d99-9b95-2c7492876e68-maas-api-service-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:23.051434 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.051413 4852 scope.go:122] "RemoveContainer" containerID="def465d755873db323adb64dc03e7c173db265996e31868fe54b66e565efd8e1" Nov 04 01:30:23.053043 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.053026 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f61cb5-7acc-4d99-9b95-2c7492876e68-kube-api-access-sgbvv" pod "03f61cb5-7acc-4d99-9b95-2c7492876e68" (UID: "03f61cb5-7acc-4d99-9b95-2c7492876e68"). InnerVolumeSpecName "kube-api-access-sgbvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:23.054136 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.054121 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a17cae9-e3d0-4a80-93ba-5602baba0d74-kube-api-access-drdsq" pod "4a17cae9-e3d0-4a80-93ba-5602baba0d74" (UID: "4a17cae9-e3d0-4a80-93ba-5602baba0d74"). InnerVolumeSpecName "kube-api-access-drdsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:23.116608 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.116346 4852 scope.go:122] "RemoveContainer" containerID="daa7ef8d4ba63ac94887500a9a7b50d3ed4851d693ab0574a516387571fa2f03" Nov 04 01:30:23.249732 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.225051 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:23.276729 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.276554 4852 scope.go:122] "RemoveContainer" containerID="27775c8ce79fed938f1194305a24696ad9b6a7f8d557472fa1b1a6d1d5dde93a" Nov 04 01:30:23.278286 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.278222 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8" Nov 04 01:30:23.400645 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.400617 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n"] Nov 04 01:30:23.406244 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.406203 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb73d5de-5e15-4453-80fc-e643b3dbd961-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb73d5de-5e15-4453-80fc-e643b3dbd961-maas-api-service-ca\") pod \"cb73d5de-5e15-4453-80fc-e643b3dbd961\" (UID: \"cb73d5de-5e15-4453-80fc-e643b3dbd961\") " Nov 04 01:30:23.406348 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.406299 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48244ae6-4584-4874-b776-610df53ed021-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48244ae6-4584-4874-b776-610df53ed021-maas-api-service-ca\") pod \"48244ae6-4584-4874-b776-610df53ed021\" (UID: \"48244ae6-4584-4874-b776-610df53ed021\") " Nov 04 01:30:23.406440 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.406415 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48244ae6-4584-4874-b776-610df53ed021-kube-api-access-rm6xj\" (UniqueName: \"kubernetes.io/projected/48244ae6-4584-4874-b776-610df53ed021-kube-api-access-rm6xj\") pod \"48244ae6-4584-4874-b776-610df53ed021\" (UID: \"48244ae6-4584-4874-b776-610df53ed021\") " Nov 04 01:30:23.407182 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.407106 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb73d5de-5e15-4453-80fc-e643b3dbd961-maas-api-service-ca" pod "cb73d5de-5e15-4453-80fc-e643b3dbd961" (UID: "cb73d5de-5e15-4453-80fc-e643b3dbd961"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:23.407182 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.407139 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48244ae6-4584-4874-b776-610df53ed021-maas-api-service-ca" pod "48244ae6-4584-4874-b776-610df53ed021" (UID: "48244ae6-4584-4874-b776-610df53ed021"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:23.407358 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.407193 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb73d5de-5e15-4453-80fc-e643b3dbd961-kube-api-access-qrfzc\" (UniqueName: \"kubernetes.io/projected/cb73d5de-5e15-4453-80fc-e643b3dbd961-kube-api-access-qrfzc\") pod \"cb73d5de-5e15-4453-80fc-e643b3dbd961\" (UID: \"cb73d5de-5e15-4453-80fc-e643b3dbd961\") " Nov 04 01:30:23.410182 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.409788 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgbvv\" (UniqueName: \"kubernetes.io/projected/03f61cb5-7acc-4d99-9b95-2c7492876e68-kube-api-access-sgbvv\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:23.410182 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.409826 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb73d5de-5e15-4453-80fc-e643b3dbd961-maas-api-service-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:23.410182 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.409841 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48244ae6-4584-4874-b776-610df53ed021-maas-api-service-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:23.410182 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.409856 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drdsq\" (UniqueName: \"kubernetes.io/projected/4a17cae9-e3d0-4a80-93ba-5602baba0d74-kube-api-access-drdsq\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:23.415837 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.413185 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb73d5de-5e15-4453-80fc-e643b3dbd961-kube-api-access-qrfzc" pod "cb73d5de-5e15-4453-80fc-e643b3dbd961" (UID: "cb73d5de-5e15-4453-80fc-e643b3dbd961"). InnerVolumeSpecName "kube-api-access-qrfzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:23.415837 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.414157 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dfcebf-29895930-g5s7n"] Nov 04 01:30:23.421245 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.421216 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48244ae6-4584-4874-b776-610df53ed021-kube-api-access-rm6xj" pod "48244ae6-4584-4874-b776-610df53ed021" (UID: "48244ae6-4584-4874-b776-610df53ed021"). InnerVolumeSpecName "kube-api-access-rm6xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:23.423684 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.423168 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4"] Nov 04 01:30:23.432548 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.432526 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-586541-29895930-cv7k4"] Nov 04 01:30:23.440690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.440668 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7" Nov 04 01:30:23.445692 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.445676 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n"] Nov 04 01:30:23.489994 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.489963 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-75f7ae-29895930-fq48n"] Nov 04 01:30:23.494263 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.494237 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5"] Nov 04 01:30:23.498091 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.498068 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-099dd9-29895930-tc9n5"] Nov 04 01:30:23.511291 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.511271 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2788c94-d82f-4ac6-94db-71ff7aeec55d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2788c94-d82f-4ac6-94db-71ff7aeec55d-maas-api-service-ca\") pod \"b2788c94-d82f-4ac6-94db-71ff7aeec55d\" (UID: \"b2788c94-d82f-4ac6-94db-71ff7aeec55d\") " Nov 04 01:30:23.511415 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.511312 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2788c94-d82f-4ac6-94db-71ff7aeec55d-kube-api-access-qkfs7\" (UniqueName: \"kubernetes.io/projected/b2788c94-d82f-4ac6-94db-71ff7aeec55d-kube-api-access-qkfs7\") pod \"b2788c94-d82f-4ac6-94db-71ff7aeec55d\" (UID: \"b2788c94-d82f-4ac6-94db-71ff7aeec55d\") " Nov 04 01:30:23.512042 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.512022 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2788c94-d82f-4ac6-94db-71ff7aeec55d-maas-api-service-ca" pod "b2788c94-d82f-4ac6-94db-71ff7aeec55d" (UID: "b2788c94-d82f-4ac6-94db-71ff7aeec55d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:23.512456 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.512437 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2788c94-d82f-4ac6-94db-71ff7aeec55d-maas-api-service-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:23.512556 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.512462 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrfzc\" (UniqueName: \"kubernetes.io/projected/cb73d5de-5e15-4453-80fc-e643b3dbd961-kube-api-access-qrfzc\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:23.512556 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.512477 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm6xj\" (UniqueName: \"kubernetes.io/projected/48244ae6-4584-4874-b776-610df53ed021-kube-api-access-rm6xj\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:23.517713 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.517685 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2788c94-d82f-4ac6-94db-71ff7aeec55d-kube-api-access-qkfs7" pod "b2788c94-d82f-4ac6-94db-71ff7aeec55d" (UID: "b2788c94-d82f-4ac6-94db-71ff7aeec55d"). InnerVolumeSpecName "kube-api-access-qkfs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:23.614243 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.614216 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkfs7\" (UniqueName: \"kubernetes.io/projected/b2788c94-d82f-4ac6-94db-71ff7aeec55d-kube-api-access-qkfs7\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:23.957493 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:23.957400 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:24.089591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:24.088874 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8" Nov 04 01:30:24.125656 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:24.125631 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7" Nov 04 01:30:24.131129 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:24.131102 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh" Nov 04 01:30:24.133941 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:24.133917 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03f61cb5-7acc-4d99-9b95-2c7492876e68" path="/var/lib/kubelet/pods/03f61cb5-7acc-4d99-9b95-2c7492876e68/volumes" Nov 04 01:30:24.134530 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:24.134509 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a17cae9-e3d0-4a80-93ba-5602baba0d74" path="/var/lib/kubelet/pods/4a17cae9-e3d0-4a80-93ba-5602baba0d74/volumes" Nov 04 01:30:24.135001 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:24.134986 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1f011e7-dc3c-4b35-8c59-76bd4101237c" path="/var/lib/kubelet/pods/c1f011e7-dc3c-4b35-8c59-76bd4101237c/volumes" Nov 04 01:30:24.135724 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:24.135708 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc68a724-a4bd-436b-92ff-4433c18960f2" path="/var/lib/kubelet/pods/dc68a724-a4bd-436b-92ff-4433c18960f2/volumes" Nov 04 01:30:24.136632 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:24.136609 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8" event={"ID":"cb73d5de-5e15-4453-80fc-e643b3dbd961","Type":"ContainerDied","Data":"18648782cd350c5ce9637d34b704cbd99b7afb1b136fb0edd6253bb796819e5e"} Nov 04 01:30:24.136754 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:24.136739 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7" event={"ID":"b2788c94-d82f-4ac6-94db-71ff7aeec55d","Type":"ContainerDied","Data":"71643a6ead70b7441a07d5d97fccf286af57d24872c3ac3b9c0cdbcc55ecdea3"} Nov 04 01:30:24.136874 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:24.136860 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh" event={"ID":"48244ae6-4584-4874-b776-610df53ed021","Type":"ContainerDied","Data":"eeb434ec0766fe9dd8842f5b1210a2fb191fe42398c506a26b455fd59dc7539d"} Nov 04 01:30:24.136971 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:24.136763 4852 scope.go:122] "RemoveContainer" containerID="b8476269693139c9ff75cc64facf9c8947cdbf71341d3a28ae724b710b7ec94a" Nov 04 01:30:24.152224 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:24.152204 4852 scope.go:122] "RemoveContainer" containerID="8658bf81b3e3c0b67eb030fa16890a473bd4e652a78a7650661df7abb961db84" Nov 04 01:30:24.184777 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:24.166769 4852 scope.go:122] "RemoveContainer" containerID="cfe6f7fb10df0aab58e506611a405a285ca3978aa680b16e88d378c89cfe8bf1" Nov 04 01:30:24.188104 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:24.188058 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8"] Nov 04 01:30:24.192061 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:24.192040 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-98f6f6-29895930-78lz8"] Nov 04 01:30:24.210889 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:24.210840 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh"] Nov 04 01:30:24.221430 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:24.221410 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-499ddd-29895930-cfjmh"] Nov 04 01:30:24.251233 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:24.251212 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7"] Nov 04 01:30:24.285065 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:24.285039 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fff32-29895930-f99t7"] Nov 04 01:30:25.684992 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:25.684959 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:25.997679 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:25.997607 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48244ae6-4584-4874-b776-610df53ed021" path="/var/lib/kubelet/pods/48244ae6-4584-4874-b776-610df53ed021/volumes" Nov 04 01:30:25.998219 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:25.998199 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2788c94-d82f-4ac6-94db-71ff7aeec55d" path="/var/lib/kubelet/pods/b2788c94-d82f-4ac6-94db-71ff7aeec55d/volumes" Nov 04 01:30:25.998689 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:25.998675 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb73d5de-5e15-4453-80fc-e643b3dbd961" path="/var/lib/kubelet/pods/cb73d5de-5e15-4453-80fc-e643b3dbd961/volumes" Nov 04 01:30:26.530586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:26.530075 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:27.402000 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:27.401900 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:28.178915 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.178879 4852 generic.go:356] "Generic (PLEG): container finished" podID="ad7e7c06-6bfc-4fee-a35a-43b73e501458" containerID="a8194384632e59c0650c08d6e51a91c98c567f33611eb537e3874f6bc5edba97" exitCode=0 Nov 04 01:30:28.179252 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.179096 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" event={"ID":"ad7e7c06-6bfc-4fee-a35a-43b73e501458","Type":"ContainerDied","Data":"a8194384632e59c0650c08d6e51a91c98c567f33611eb537e3874f6bc5edba97"} Nov 04 01:30:28.195539 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.195295 4852 generic.go:356] "Generic (PLEG): container finished" podID="3a4d4cee-bd81-4a99-8d45-c636a02c64cf" containerID="435f0eb11db1abdba93331fd4fcda004baef191174a6fd5edfc3d881c235b2c4" exitCode=0 Nov 04 01:30:28.195539 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.195456 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" event={"ID":"3a4d4cee-bd81-4a99-8d45-c636a02c64cf","Type":"ContainerDied","Data":"435f0eb11db1abdba93331fd4fcda004baef191174a6fd5edfc3d881c235b2c4"} Nov 04 01:30:28.198008 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.197896 4852 generic.go:356] "Generic (PLEG): container finished" podID="fbc69a29-d23c-4da4-a6af-be04977ee457" containerID="3fd939b9ff86004e1acc4c88457314d2ca6faa02cbc7e28c2b59eacfe8bb2613" exitCode=0 Nov 04 01:30:28.198008 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.197961 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" event={"ID":"fbc69a29-d23c-4da4-a6af-be04977ee457","Type":"ContainerDied","Data":"3fd939b9ff86004e1acc4c88457314d2ca6faa02cbc7e28c2b59eacfe8bb2613"} Nov 04 01:30:28.308536 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.308495 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:28.553695 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.541019 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:30:28.553695 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.541925 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:28.577130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.576407 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbc69a29-d23c-4da4-a6af-be04977ee457-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbc69a29-d23c-4da4-a6af-be04977ee457-istiod-ca-cert\") pod \"fbc69a29-d23c-4da4-a6af-be04977ee457\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " Nov 04 01:30:28.577130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.576521 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-envoy\") pod \"fbc69a29-d23c-4da4-a6af-be04977ee457\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " Nov 04 01:30:28.577130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.576531 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:30:28.577130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.576624 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-data\") pod \"fbc69a29-d23c-4da4-a6af-be04977ee457\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " Nov 04 01:30:28.577130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.576688 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbc69a29-d23c-4da4-a6af-be04977ee457-kube-api-access-s79x4\" (UniqueName: \"kubernetes.io/projected/fbc69a29-d23c-4da4-a6af-be04977ee457-kube-api-access-s79x4\") pod \"fbc69a29-d23c-4da4-a6af-be04977ee457\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " Nov 04 01:30:28.577130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.576714 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-podinfo\") pod \"fbc69a29-d23c-4da4-a6af-be04977ee457\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " Nov 04 01:30:28.577130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.576827 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-workload-certs\") pod \"fbc69a29-d23c-4da4-a6af-be04977ee457\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " Nov 04 01:30:28.577130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.576969 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-token\" (UniqueName: \"kubernetes.io/projected/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-token\") pod \"fbc69a29-d23c-4da4-a6af-be04977ee457\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " Nov 04 01:30:28.577130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.577030 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-credential-socket\") pod \"fbc69a29-d23c-4da4-a6af-be04977ee457\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " Nov 04 01:30:28.577783 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.577135 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-workload-socket\") pod \"fbc69a29-d23c-4da4-a6af-be04977ee457\" (UID: \"fbc69a29-d23c-4da4-a6af-be04977ee457\") " Nov 04 01:30:28.578165 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.577948 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-data" pod "fbc69a29-d23c-4da4-a6af-be04977ee457" (UID: "fbc69a29-d23c-4da4-a6af-be04977ee457"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:28.578286 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.578178 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-workload-socket" pod "fbc69a29-d23c-4da4-a6af-be04977ee457" (UID: "fbc69a29-d23c-4da4-a6af-be04977ee457"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:28.578801 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.578483 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbc69a29-d23c-4da4-a6af-be04977ee457-istiod-ca-cert" pod "fbc69a29-d23c-4da4-a6af-be04977ee457" (UID: "fbc69a29-d23c-4da4-a6af-be04977ee457"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:28.578801 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.578527 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-workload-certs" pod "fbc69a29-d23c-4da4-a6af-be04977ee457" (UID: "fbc69a29-d23c-4da4-a6af-be04977ee457"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:28.578801 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.578553 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.578801 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.578593 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.579201 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.579139 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-credential-socket" pod "fbc69a29-d23c-4da4-a6af-be04977ee457" (UID: "fbc69a29-d23c-4da4-a6af-be04977ee457"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:28.582752 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.582706 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-podinfo" pod "fbc69a29-d23c-4da4-a6af-be04977ee457" (UID: "fbc69a29-d23c-4da4-a6af-be04977ee457"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:28.582945 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.582854 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc69a29-d23c-4da4-a6af-be04977ee457-kube-api-access-s79x4" pod "fbc69a29-d23c-4da4-a6af-be04977ee457" (UID: "fbc69a29-d23c-4da4-a6af-be04977ee457"). InnerVolumeSpecName "kube-api-access-s79x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:28.584064 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.584021 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-envoy" pod "fbc69a29-d23c-4da4-a6af-be04977ee457" (UID: "fbc69a29-d23c-4da4-a6af-be04977ee457"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:28.584185 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.584097 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-token" pod "fbc69a29-d23c-4da4-a6af-be04977ee457" (UID: "fbc69a29-d23c-4da4-a6af-be04977ee457"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:28.685326 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.685287 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-envoy\") pod \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " Nov 04 01:30:28.685514 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.685404 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-workload-socket\") pod \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " Nov 04 01:30:28.685514 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.685454 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istiod-ca-cert\") pod \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " Nov 04 01:30:28.685514 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.685506 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-data\") pod \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " Nov 04 01:30:28.685703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.685552 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-kube-api-access-4sw5d\" (UniqueName: \"kubernetes.io/projected/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-kube-api-access-4sw5d\") pod \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " Nov 04 01:30:28.685703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.685688 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-credential-socket\") pod \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " Nov 04 01:30:28.685849 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.685832 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-workload-certs\") pod \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " Nov 04 01:30:28.685907 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.685869 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-podinfo\") pod \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " Nov 04 01:30:28.685962 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.685912 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-token\" (UniqueName: \"kubernetes.io/projected/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-token\") pod \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\" (UID: \"3a4d4cee-bd81-4a99-8d45-c636a02c64cf\") " Nov 04 01:30:28.686823 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.686797 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbc69a29-d23c-4da4-a6af-be04977ee457-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.686939 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.686825 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.686939 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.686841 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s79x4\" (UniqueName: \"kubernetes.io/projected/fbc69a29-d23c-4da4-a6af-be04977ee457-kube-api-access-s79x4\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.686939 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.686856 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.686939 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.686871 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.686939 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.686884 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbc69a29-d23c-4da4-a6af-be04977ee457-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.686939 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.686898 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc69a29-d23c-4da4-a6af-be04977ee457-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.687228 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.687162 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-workload-socket" pod "3a4d4cee-bd81-4a99-8d45-c636a02c64cf" (UID: "3a4d4cee-bd81-4a99-8d45-c636a02c64cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:28.687613 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.687546 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istiod-ca-cert" pod "3a4d4cee-bd81-4a99-8d45-c636a02c64cf" (UID: "3a4d4cee-bd81-4a99-8d45-c636a02c64cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:28.687848 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.687827 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-data" pod "3a4d4cee-bd81-4a99-8d45-c636a02c64cf" (UID: "3a4d4cee-bd81-4a99-8d45-c636a02c64cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:28.690231 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.690206 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-workload-certs" pod "3a4d4cee-bd81-4a99-8d45-c636a02c64cf" (UID: "3a4d4cee-bd81-4a99-8d45-c636a02c64cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:28.690553 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.690532 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-credential-socket" pod "3a4d4cee-bd81-4a99-8d45-c636a02c64cf" (UID: "3a4d4cee-bd81-4a99-8d45-c636a02c64cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:28.691079 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.691042 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-kube-api-access-4sw5d" pod "3a4d4cee-bd81-4a99-8d45-c636a02c64cf" (UID: "3a4d4cee-bd81-4a99-8d45-c636a02c64cf"). InnerVolumeSpecName "kube-api-access-4sw5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:28.695062 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.694966 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-token" pod "3a4d4cee-bd81-4a99-8d45-c636a02c64cf" (UID: "3a4d4cee-bd81-4a99-8d45-c636a02c64cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:28.701489 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.701463 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-envoy" pod "3a4d4cee-bd81-4a99-8d45-c636a02c64cf" (UID: "3a4d4cee-bd81-4a99-8d45-c636a02c64cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:28.705347 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.705320 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-podinfo" pod "3a4d4cee-bd81-4a99-8d45-c636a02c64cf" (UID: "3a4d4cee-bd81-4a99-8d45-c636a02c64cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:28.706805 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.706783 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:30:28.790323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.788408 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-token\" (UniqueName: \"kubernetes.io/projected/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-token\") pod \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " Nov 04 01:30:28.790323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.788505 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-envoy\") pod \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " Nov 04 01:30:28.790323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.788667 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-credential-socket\") pod \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " Nov 04 01:30:28.790323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.788735 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-data\") pod \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " Nov 04 01:30:28.790323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.788855 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-workload-certs\") pod \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " Nov 04 01:30:28.790323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.788892 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-workload-socket\") pod \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " Nov 04 01:30:28.790323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.788983 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istiod-ca-cert\") pod \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " Nov 04 01:30:28.790323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.789054 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-podinfo\") pod \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " Nov 04 01:30:28.790323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.789102 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad7e7c06-6bfc-4fee-a35a-43b73e501458-kube-api-access-wx6p8\" (UniqueName: \"kubernetes.io/projected/ad7e7c06-6bfc-4fee-a35a-43b73e501458-kube-api-access-wx6p8\") pod \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\" (UID: \"ad7e7c06-6bfc-4fee-a35a-43b73e501458\") " Nov 04 01:30:28.790323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.789903 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.790323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.789927 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.790323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.789940 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.790323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.789949 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.790323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.789958 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sw5d\" (UniqueName: \"kubernetes.io/projected/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-kube-api-access-4sw5d\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.790323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.789970 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.790323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.789982 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.790323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.789992 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.790323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.790006 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a4d4cee-bd81-4a99-8d45-c636a02c64cf-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.791857 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.791812 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-workload-certs" pod "ad7e7c06-6bfc-4fee-a35a-43b73e501458" (UID: "ad7e7c06-6bfc-4fee-a35a-43b73e501458"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:28.792239 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.792095 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istiod-ca-cert" pod "ad7e7c06-6bfc-4fee-a35a-43b73e501458" (UID: "ad7e7c06-6bfc-4fee-a35a-43b73e501458"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:28.792239 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.792188 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-workload-socket" pod "ad7e7c06-6bfc-4fee-a35a-43b73e501458" (UID: "ad7e7c06-6bfc-4fee-a35a-43b73e501458"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:28.792704 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.792663 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-credential-socket" pod "ad7e7c06-6bfc-4fee-a35a-43b73e501458" (UID: "ad7e7c06-6bfc-4fee-a35a-43b73e501458"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:28.793214 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.793186 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-data" pod "ad7e7c06-6bfc-4fee-a35a-43b73e501458" (UID: "ad7e7c06-6bfc-4fee-a35a-43b73e501458"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:28.794233 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.794201 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad7e7c06-6bfc-4fee-a35a-43b73e501458-kube-api-access-wx6p8" pod "ad7e7c06-6bfc-4fee-a35a-43b73e501458" (UID: "ad7e7c06-6bfc-4fee-a35a-43b73e501458"). InnerVolumeSpecName "kube-api-access-wx6p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:28.796117 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.796092 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-envoy" pod "ad7e7c06-6bfc-4fee-a35a-43b73e501458" (UID: "ad7e7c06-6bfc-4fee-a35a-43b73e501458"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:28.796117 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.796102 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-token" pod "ad7e7c06-6bfc-4fee-a35a-43b73e501458" (UID: "ad7e7c06-6bfc-4fee-a35a-43b73e501458"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:28.796710 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.796684 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-podinfo" pod "ad7e7c06-6bfc-4fee-a35a-43b73e501458" (UID: "ad7e7c06-6bfc-4fee-a35a-43b73e501458"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:28.891824 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.891736 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.891824 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.891776 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.891824 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.891792 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx6p8\" (UniqueName: \"kubernetes.io/projected/ad7e7c06-6bfc-4fee-a35a-43b73e501458-kube-api-access-wx6p8\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.891824 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.891805 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.891824 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.891818 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.891824 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.891828 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.892198 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.891842 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.892198 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.891857 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.892198 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:28.891871 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7e7c06-6bfc-4fee-a35a-43b73e501458-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:29.263654 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:29.263621 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" event={"ID":"ad7e7c06-6bfc-4fee-a35a-43b73e501458","Type":"ContainerDied","Data":"e8929607325bf07e0f390a17029713285d711e5905f266761f77b542986d2fd3"} Nov 04 01:30:29.263888 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:29.263873 4852 scope.go:122] "RemoveContainer" containerID="a8194384632e59c0650c08d6e51a91c98c567f33611eb537e3874f6bc5edba97" Nov 04 01:30:29.264162 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:29.264145 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp" Nov 04 01:30:29.311602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:29.305453 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" event={"ID":"3a4d4cee-bd81-4a99-8d45-c636a02c64cf","Type":"ContainerDied","Data":"eaea6aad4e9be4307e03033629195395ca4d649b845ef8b0fa12fa4e68c4b6d1"} Nov 04 01:30:29.311602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:29.305677 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl" Nov 04 01:30:29.361883 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:29.361849 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" event={"ID":"fbc69a29-d23c-4da4-a6af-be04977ee457","Type":"ContainerDied","Data":"5540814cf0403bf2a06acfe1eed949afc965567043e2a451c1fbe5f6f1edd0aa"} Nov 04 01:30:29.362093 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:29.361983 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d" Nov 04 01:30:29.383625 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:29.383602 4852 scope.go:122] "RemoveContainer" containerID="435f0eb11db1abdba93331fd4fcda004baef191174a6fd5edfc3d881c235b2c4" Nov 04 01:30:29.394807 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:29.394787 4852 scope.go:122] "RemoveContainer" containerID="3fd939b9ff86004e1acc4c88457314d2ca6faa02cbc7e28c2b59eacfe8bb2613" Nov 04 01:30:29.420966 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:29.420936 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp"] Nov 04 01:30:29.431291 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:29.431265 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-s8wxp"] Nov 04 01:30:29.443143 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:29.443118 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl"] Nov 04 01:30:29.451165 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:29.451136 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-q2vvl"] Nov 04 01:30:29.470500 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:29.470472 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d"] Nov 04 01:30:29.477128 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:29.477100 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-zc25d"] Nov 04 01:30:29.631317 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:29.631193 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:30.089989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:30.089822 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a4d4cee-bd81-4a99-8d45-c636a02c64cf" path="/var/lib/kubelet/pods/3a4d4cee-bd81-4a99-8d45-c636a02c64cf/volumes" Nov 04 01:30:30.091031 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:30.091002 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad7e7c06-6bfc-4fee-a35a-43b73e501458" path="/var/lib/kubelet/pods/ad7e7c06-6bfc-4fee-a35a-43b73e501458/volumes" Nov 04 01:30:30.092655 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:30.092337 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbc69a29-d23c-4da4-a6af-be04977ee457" path="/var/lib/kubelet/pods/fbc69a29-d23c-4da4-a6af-be04977ee457/volumes" Nov 04 01:30:30.108139 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:30:30.102533 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:30:30.670600 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:30.664073 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:31.361616 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.361444 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:31.918398 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.918362 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb"] Nov 04 01:30:31.920020 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.919982 4852 cpu_manager.go:414] "containerMap: removing container" podUID="d9585268-ca61-4539-84d5-a986d3161a1d" containerName="cleanup" Nov 04 01:30:31.920183 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920155 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9585268-ca61-4539-84d5-a986d3161a1d" containerName="cleanup" Nov 04 01:30:31.920323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920192 4852 cpu_manager.go:414] "containerMap: removing container" podUID="48244ae6-4584-4874-b776-610df53ed021" containerName="cleanup" Nov 04 01:30:31.920323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920200 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48244ae6-4584-4874-b776-610df53ed021" containerName="cleanup" Nov 04 01:30:31.920323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920213 4852 cpu_manager.go:414] "containerMap: removing container" podUID="b2788c94-d82f-4ac6-94db-71ff7aeec55d" containerName="cleanup" Nov 04 01:30:31.920323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920219 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2788c94-d82f-4ac6-94db-71ff7aeec55d" containerName="cleanup" Nov 04 01:30:31.920323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920229 4852 cpu_manager.go:414] "containerMap: removing container" podUID="4a17cae9-e3d0-4a80-93ba-5602baba0d74" containerName="cleanup" Nov 04 01:30:31.920323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920236 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a17cae9-e3d0-4a80-93ba-5602baba0d74" containerName="cleanup" Nov 04 01:30:31.920323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920253 4852 cpu_manager.go:414] "containerMap: removing container" podUID="cb73d5de-5e15-4453-80fc-e643b3dbd961" containerName="cleanup" Nov 04 01:30:31.920323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920259 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb73d5de-5e15-4453-80fc-e643b3dbd961" containerName="cleanup" Nov 04 01:30:31.920323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920270 4852 cpu_manager.go:414] "containerMap: removing container" podUID="03f61cb5-7acc-4d99-9b95-2c7492876e68" containerName="cleanup" Nov 04 01:30:31.920323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920277 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03f61cb5-7acc-4d99-9b95-2c7492876e68" containerName="cleanup" Nov 04 01:30:31.920323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920295 4852 cpu_manager.go:414] "containerMap: removing container" podUID="dc68a724-a4bd-436b-92ff-4433c18960f2" containerName="cleanup" Nov 04 01:30:31.920323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920301 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc68a724-a4bd-436b-92ff-4433c18960f2" containerName="cleanup" Nov 04 01:30:31.920323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920317 4852 cpu_manager.go:414] "containerMap: removing container" podUID="fbc69a29-d23c-4da4-a6af-be04977ee457" containerName="istio-proxy" Nov 04 01:30:31.920323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920323 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc69a29-d23c-4da4-a6af-be04977ee457" containerName="istio-proxy" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920336 4852 cpu_manager.go:414] "containerMap: removing container" podUID="ad7e7c06-6bfc-4fee-a35a-43b73e501458" containerName="istio-proxy" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920343 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad7e7c06-6bfc-4fee-a35a-43b73e501458" containerName="istio-proxy" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920358 4852 cpu_manager.go:414] "containerMap: removing container" podUID="b2788c94-d82f-4ac6-94db-71ff7aeec55d" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920364 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2788c94-d82f-4ac6-94db-71ff7aeec55d" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920375 4852 cpu_manager.go:414] "containerMap: removing container" podUID="b2788c94-d82f-4ac6-94db-71ff7aeec55d" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920381 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2788c94-d82f-4ac6-94db-71ff7aeec55d" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920398 4852 cpu_manager.go:414] "containerMap: removing container" podUID="cb73d5de-5e15-4453-80fc-e643b3dbd961" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920405 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb73d5de-5e15-4453-80fc-e643b3dbd961" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920414 4852 cpu_manager.go:414] "containerMap: removing container" podUID="03f61cb5-7acc-4d99-9b95-2c7492876e68" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920420 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03f61cb5-7acc-4d99-9b95-2c7492876e68" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920436 4852 cpu_manager.go:414] "containerMap: removing container" podUID="c1f011e7-dc3c-4b35-8c59-76bd4101237c" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920442 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f011e7-dc3c-4b35-8c59-76bd4101237c" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920456 4852 cpu_manager.go:414] "containerMap: removing container" podUID="c1f011e7-dc3c-4b35-8c59-76bd4101237c" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920463 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f011e7-dc3c-4b35-8c59-76bd4101237c" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920473 4852 cpu_manager.go:414] "containerMap: removing container" podUID="c1f011e7-dc3c-4b35-8c59-76bd4101237c" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920480 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f011e7-dc3c-4b35-8c59-76bd4101237c" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920493 4852 cpu_manager.go:414] "containerMap: removing container" podUID="48244ae6-4584-4874-b776-610df53ed021" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920500 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48244ae6-4584-4874-b776-610df53ed021" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920511 4852 cpu_manager.go:414] "containerMap: removing container" podUID="3a4d4cee-bd81-4a99-8d45-c636a02c64cf" containerName="istio-proxy" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920518 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a4d4cee-bd81-4a99-8d45-c636a02c64cf" containerName="istio-proxy" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920527 4852 cpu_manager.go:414] "containerMap: removing container" podUID="48244ae6-4584-4874-b776-610df53ed021" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920533 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48244ae6-4584-4874-b776-610df53ed021" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920546 4852 cpu_manager.go:414] "containerMap: removing container" podUID="dc68a724-a4bd-436b-92ff-4433c18960f2" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920552 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc68a724-a4bd-436b-92ff-4433c18960f2" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920579 4852 cpu_manager.go:414] "containerMap: removing container" podUID="4a17cae9-e3d0-4a80-93ba-5602baba0d74" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920585 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a17cae9-e3d0-4a80-93ba-5602baba0d74" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920740 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc68a724-a4bd-436b-92ff-4433c18960f2" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920752 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb73d5de-5e15-4453-80fc-e643b3dbd961" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920759 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="03f61cb5-7acc-4d99-9b95-2c7492876e68" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920768 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f011e7-dc3c-4b35-8c59-76bd4101237c" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920777 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f011e7-dc3c-4b35-8c59-76bd4101237c" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920785 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f011e7-dc3c-4b35-8c59-76bd4101237c" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920794 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="48244ae6-4584-4874-b776-610df53ed021" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920801 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a4d4cee-bd81-4a99-8d45-c636a02c64cf" containerName="istio-proxy" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920811 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="48244ae6-4584-4874-b776-610df53ed021" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920820 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="03f61cb5-7acc-4d99-9b95-2c7492876e68" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920827 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc68a724-a4bd-436b-92ff-4433c18960f2" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920833 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a17cae9-e3d0-4a80-93ba-5602baba0d74" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920843 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9585268-ca61-4539-84d5-a986d3161a1d" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920851 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="48244ae6-4584-4874-b776-610df53ed021" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920858 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2788c94-d82f-4ac6-94db-71ff7aeec55d" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920864 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a17cae9-e3d0-4a80-93ba-5602baba0d74" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920873 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb73d5de-5e15-4453-80fc-e643b3dbd961" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920880 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a17cae9-e3d0-4a80-93ba-5602baba0d74" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920888 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb73d5de-5e15-4453-80fc-e643b3dbd961" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920895 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9585268-ca61-4539-84d5-a986d3161a1d" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920904 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbc69a29-d23c-4da4-a6af-be04977ee457" containerName="istio-proxy" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920914 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad7e7c06-6bfc-4fee-a35a-43b73e501458" containerName="istio-proxy" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920923 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2788c94-d82f-4ac6-94db-71ff7aeec55d" containerName="cleanup" Nov 04 01:30:31.921138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.920930 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2788c94-d82f-4ac6-94db-71ff7aeec55d" containerName="cleanup" Nov 04 01:30:31.946609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.938880 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:31.946609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:31.946504 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb"] Nov 04 01:30:32.008276 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.008244 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.008423 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.008322 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f24a4b-278a-432f-a86b-f3addc378bee-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.008474 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.008448 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.008531 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.008506 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f24a4b-278a-432f-a86b-f3addc378bee-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.008676 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.008653 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.008830 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.008815 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.008898 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.008871 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.008995 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.008981 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f24a4b-278a-432f-a86b-f3addc378bee-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.009065 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.009038 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrjlw\" (UniqueName: \"kubernetes.io/projected/85f24a4b-278a-432f-a86b-f3addc378bee-kube-api-access-xrjlw\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.117294 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.115629 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.117294 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.115713 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.117294 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.116423 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f24a4b-278a-432f-a86b-f3addc378bee-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.117294 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.116502 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrjlw\" (UniqueName: \"kubernetes.io/projected/85f24a4b-278a-432f-a86b-f3addc378bee-kube-api-access-xrjlw\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.117294 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.116633 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.117294 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.116700 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f24a4b-278a-432f-a86b-f3addc378bee-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.117294 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.116769 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.117294 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.116817 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f24a4b-278a-432f-a86b-f3addc378bee-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.117294 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.116902 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.118616 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.118539 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.119090 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.119066 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.129609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.124834 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.129609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.125655 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f24a4b-278a-432f-a86b-f3addc378bee-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.129609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.126260 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.132072 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.132043 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:32.132398 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.132373 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f24a4b-278a-432f-a86b-f3addc378bee-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.135250 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.135218 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.136515 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.136470 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f24a4b-278a-432f-a86b-f3addc378bee-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.136707 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.136579 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrjlw\" (UniqueName: \"kubernetes.io/projected/85f24a4b-278a-432f-a86b-f3addc378bee-kube-api-access-xrjlw\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.262784 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.262752 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:32.470056 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.470029 4852 generic.go:356] "Generic (PLEG): container finished" podID="57537b5d-473c-41da-9e67-6c848a66fee7" containerID="0205cb769edf06deebde37199e8ff5561c058fc76dbd456b0b36b62c04e5360e" exitCode=0 Nov 04 01:30:32.470201 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.470097 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" event={"ID":"57537b5d-473c-41da-9e67-6c848a66fee7","Type":"ContainerDied","Data":"0205cb769edf06deebde37199e8ff5561c058fc76dbd456b0b36b62c04e5360e"} Nov 04 01:30:32.470607 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.470585 4852 scope.go:122] "RemoveContainer" containerID="0205cb769edf06deebde37199e8ff5561c058fc76dbd456b0b36b62c04e5360e" Nov 04 01:30:32.472407 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.472379 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:32.472485 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.472444 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:32.472485 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.472475 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:32.575716 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.575691 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb"] Nov 04 01:30:32.578972 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:30:32.578946 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f24a4b_278a_432f_a86b_f3addc378bee.slice/crio-e06b4a783ba3ee22e9825f8c2988a63417b395cf486b3d9865cf8aff4d57a4b7 WatchSource:0}: Error finding container e06b4a783ba3ee22e9825f8c2988a63417b395cf486b3d9865cf8aff4d57a4b7: Status 404 returned error can't find the container with id e06b4a783ba3ee22e9825f8c2988a63417b395cf486b3d9865cf8aff4d57a4b7 Nov 04 01:30:32.582355 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.582322 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:32.582444 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.582390 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:32.582444 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.582423 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:32.893642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:32.891443 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:33.510545 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:33.510502 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" event={"ID":"85f24a4b-278a-432f-a86b-f3addc378bee","Type":"ContainerStarted","Data":"4662529fdebde9b66f372fcfbce7255f8677c0f42baa49b09de1c50d0217ebce"} Nov 04 01:30:33.511089 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:33.511068 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" event={"ID":"85f24a4b-278a-432f-a86b-f3addc378bee","Type":"ContainerStarted","Data":"e06b4a783ba3ee22e9825f8c2988a63417b395cf486b3d9865cf8aff4d57a4b7"} Nov 04 01:30:33.515671 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:33.515645 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" event={"ID":"57537b5d-473c-41da-9e67-6c848a66fee7","Type":"ContainerStarted","Data":"1d6201d9c2ff6e099ef6ac5252c7c828c889c83a42af01358c2e567bb839831e"} Nov 04 01:30:33.596739 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:33.587836 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" podStartSLOduration=2.587816153 podStartE2EDuration="2.587816153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:33.555623941 +0000 UTC m=+1915.670734060" watchObservedRunningTime="2026-11-04 01:30:33.587816153 +0000 UTC m=+1915.702926271" Nov 04 01:30:33.628585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:33.618818 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:34.140085 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:34.140050 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:30:34.140303 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:34.140257 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 04 01:30:34.140383 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:34.140308 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" podUID="57537b5d-473c-41da-9e67-6c848a66fee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 04 01:30:34.302592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:34.287464 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:34.302592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:34.299964 4852 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:30:34.302592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:34.300021 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" podUID="85f24a4b-278a-432f-a86b-f3addc378bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 04 01:30:34.432273 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:34.432167 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:35.140274 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:35.140240 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 04 01:30:35.140723 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:35.140297 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" podUID="57537b5d-473c-41da-9e67-6c848a66fee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 04 01:30:35.270595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:35.267717 4852 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:30:35.270595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:35.267776 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" podUID="85f24a4b-278a-432f-a86b-f3addc378bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 04 01:30:35.281360 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:35.280811 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:36.156189 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:36.151958 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 04 01:30:36.156189 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:36.152055 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" podUID="57537b5d-473c-41da-9e67-6c848a66fee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 04 01:30:36.177315 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:36.177259 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:36.277589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:36.274664 4852 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:30:36.277589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:36.274726 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" podUID="85f24a4b-278a-432f-a86b-f3addc378bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 04 01:30:37.064592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:37.059150 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:37.147590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:37.147136 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 04 01:30:37.147590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:37.147195 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" podUID="57537b5d-473c-41da-9e67-6c848a66fee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 04 01:30:37.268396 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:37.267962 4852 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:30:37.268396 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:37.268022 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" podUID="85f24a4b-278a-432f-a86b-f3addc378bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 04 01:30:37.807198 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:37.807153 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:38.123879 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.123714 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:38.142224 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.139879 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 04 01:30:38.142224 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.139935 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" podUID="57537b5d-473c-41da-9e67-6c848a66fee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 04 01:30:38.149670 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.149643 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc"] Nov 04 01:30:38.151337 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.151313 4852 cpu_manager.go:414] "containerMap: removing container" podUID="d9585268-ca61-4539-84d5-a986d3161a1d" containerName="cleanup" Nov 04 01:30:38.151468 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.151456 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9585268-ca61-4539-84d5-a986d3161a1d" containerName="cleanup" Nov 04 01:30:38.151614 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.151557 4852 cpu_manager.go:414] "containerMap: removing container" podUID="4a17cae9-e3d0-4a80-93ba-5602baba0d74" containerName="cleanup" Nov 04 01:30:38.151704 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.151694 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a17cae9-e3d0-4a80-93ba-5602baba0d74" containerName="cleanup" Nov 04 01:30:38.151820 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.151809 4852 cpu_manager.go:414] "containerMap: removing container" podUID="dc68a724-a4bd-436b-92ff-4433c18960f2" containerName="cleanup" Nov 04 01:30:38.151897 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.151886 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc68a724-a4bd-436b-92ff-4433c18960f2" containerName="cleanup" Nov 04 01:30:38.152009 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.152001 4852 cpu_manager.go:414] "containerMap: removing container" podUID="cb73d5de-5e15-4453-80fc-e643b3dbd961" containerName="cleanup" Nov 04 01:30:38.152061 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.152054 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb73d5de-5e15-4453-80fc-e643b3dbd961" containerName="cleanup" Nov 04 01:30:38.152108 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.152102 4852 cpu_manager.go:414] "containerMap: removing container" podUID="03f61cb5-7acc-4d99-9b95-2c7492876e68" containerName="cleanup" Nov 04 01:30:38.152156 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.152149 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03f61cb5-7acc-4d99-9b95-2c7492876e68" containerName="cleanup" Nov 04 01:30:38.152637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.152611 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="03f61cb5-7acc-4d99-9b95-2c7492876e68" containerName="cleanup" Nov 04 01:30:38.152824 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.152808 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc68a724-a4bd-436b-92ff-4433c18960f2" containerName="cleanup" Nov 04 01:30:38.220266 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.220210 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc"] Nov 04 01:30:38.220497 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.220424 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.296591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.282134 4852 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:30:38.296591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.282187 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" podUID="85f24a4b-278a-432f-a86b-f3addc378bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 04 01:30:38.296591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.291498 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-envoy\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.296591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.291628 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-workload-certs\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.296591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.291667 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-podinfo\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.296591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.291705 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-credential-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.296591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.291760 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istiod-ca-cert\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.296591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.291802 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgvb6\" (UniqueName: \"kubernetes.io/projected/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-kube-api-access-kgvb6\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.296591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.291968 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-workload-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.296591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.292084 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-token\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.296591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.292139 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-data\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.407640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.398866 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-envoy\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.407640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.399005 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-workload-certs\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.407640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.399044 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-podinfo\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.407640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.399085 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-credential-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.407640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.399142 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istiod-ca-cert\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.407640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.399188 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgvb6\" (UniqueName: \"kubernetes.io/projected/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-kube-api-access-kgvb6\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.407640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.399373 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-workload-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.407640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.399498 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-token\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.407640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.399579 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-data\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.407640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.400154 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-data\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.408096 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.407976 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istiod-ca-cert\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.411589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.408218 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-workload-certs\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.411589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.409087 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-workload-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.411589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.409808 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-credential-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.411589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.410867 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-envoy\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.413092 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.413073 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-podinfo\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.418979 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.418953 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-token\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.419674 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.419654 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgvb6\" (UniqueName: \"kubernetes.io/projected/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-kube-api-access-kgvb6\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.436448 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.436422 4852 scope.go:122] "RemoveContainer" containerID="2c1cf64259645efa8913d307a0260514357b787c4e9ca38da0a6e87e0ecefec4" Nov 04 01:30:38.539945 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.539922 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:38.890588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:38.840874 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:39.140900 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:39.140866 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 04 01:30:39.141108 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:39.140916 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" podUID="57537b5d-473c-41da-9e67-6c848a66fee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 04 01:30:39.163774 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:30:39.163730 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784fa5a1_0f23_4a6e_a6c4_eae42e88f7da.slice/crio-35b0cdf71212dbb1d92997e4764faf68666596b46b55284b0532b74bed6e816d WatchSource:0}: Error finding container 35b0cdf71212dbb1d92997e4764faf68666596b46b55284b0532b74bed6e816d: Status 404 returned error can't find the container with id 35b0cdf71212dbb1d92997e4764faf68666596b46b55284b0532b74bed6e816d Nov 04 01:30:39.163992 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:39.163969 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc"] Nov 04 01:30:39.166986 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:39.166954 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:39.167085 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:39.167017 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:39.167085 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:39.167067 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:39.264483 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:39.264443 4852 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:30:39.264746 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:39.264513 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" podUID="85f24a4b-278a-432f-a86b-f3addc378bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 04 01:30:39.297301 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:39.296254 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:30:39.690555 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:39.690221 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:39.737987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:39.736198 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" event={"ID":"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da","Type":"ContainerStarted","Data":"005f8df3f988624bff36a057c00e0df6e8c3181b72b3efe39c1913e008180752"} Nov 04 01:30:39.737987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:39.736263 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" event={"ID":"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da","Type":"ContainerStarted","Data":"35b0cdf71212dbb1d92997e4764faf68666596b46b55284b0532b74bed6e816d"} Nov 04 01:30:39.786502 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:39.786447 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" podStartSLOduration=2.78643279 podStartE2EDuration="2.78643279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:39.785767931 +0000 UTC m=+1921.900878053" watchObservedRunningTime="2026-11-04 01:30:39.78643279 +0000 UTC m=+1921.901542907" Nov 04 01:30:40.144771 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:40.144744 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:30:40.145933 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:40.145911 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:30:40.287666 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:40.287635 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:40.288093 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:40.288070 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:40.289116 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:40.289097 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:30:40.541153 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:40.541120 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:40.542401 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:40.542377 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:30:40.542544 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:40.542433 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" podUID="784fa5a1-0f23-4a6e-a6c4-eae42e88f7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 04 01:30:40.725105 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:40.725075 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:41.541058 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:41.541024 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:30:41.541259 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:41.541094 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" podUID="784fa5a1-0f23-4a6e-a6c4-eae42e88f7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 04 01:30:41.686258 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:41.685960 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:42.550587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:42.549999 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:30:42.550587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:42.550059 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" podUID="784fa5a1-0f23-4a6e-a6c4-eae42e88f7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 04 01:30:42.587627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:42.584470 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:43.423027 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:43.419833 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:43.554484 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:43.554450 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:30:43.555117 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:43.555084 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" podUID="784fa5a1-0f23-4a6e-a6c4-eae42e88f7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 04 01:30:44.548040 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:44.547996 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:30:44.548420 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:44.548384 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" podUID="784fa5a1-0f23-4a6e-a6c4-eae42e88f7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 04 01:30:44.549189 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:44.549165 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:45.430131 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:45.423667 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:45.542953 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:45.542908 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:30:45.550592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:45.543173 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" podUID="784fa5a1-0f23-4a6e-a6c4-eae42e88f7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 04 01:30:46.389586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:46.379393 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:46.545424 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:46.545396 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:46.545953 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:46.545796 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:46.546499 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:46.546477 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:30:47.366280 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:47.366183 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:48.233637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:48.231013 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:48.982296 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:48.982230 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:49.897914 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:49.897874 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:50.763380 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:50.763340 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:51.074922 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:51.074836 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:51.321941 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:51.321888 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:51.706353 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:51.706321 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:51.984645 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:51.978481 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:52.361590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:52.353042 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:52.539809 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:52.539779 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:53.272404 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:53.264543 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:54.112709 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:54.112674 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:55.378594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:55.377360 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:55.784531 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:55.784496 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:56.039631 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:56.039509 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:56.778796 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:56.778763 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:57.282249 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:57.282217 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:57.504958 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:57.504929 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:58.127592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:58.124957 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:59.354469 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:30:59.354431 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:30:59.868061 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:30:59.868025 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:31:00.179265 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:00.178856 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:00.881554 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:00.881511 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr"] Nov 04 01:31:01.000191 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:01.000146 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:01.212168 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:01.207712 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:01.410084 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:01.410051 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:01.582602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:01.582543 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:02.378472 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:02.378435 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:31:02.448595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:02.444164 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:03.345323 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:03.345172 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:04.330606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:04.326906 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:05.098020 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:05.097985 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:05.882774 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:05.882720 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr" podUID="edada0ad-6b72-442b-ade8-b4f02d71f86c" containerName="payload-pre-processing" containerID="cri-o://c0cc71a0e0185f4cfacb1f9c7143c79dd959ed165f702f2f9f5cd7d28da9fc37" gracePeriod=25 Nov 04 01:31:06.092272 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:06.092177 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:06.347967 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:06.347940 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr" Nov 04 01:31:06.396385 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:06.396348 4852 generic.go:356] "Generic (PLEG): container finished" podID="edada0ad-6b72-442b-ade8-b4f02d71f86c" containerID="c0cc71a0e0185f4cfacb1f9c7143c79dd959ed165f702f2f9f5cd7d28da9fc37" exitCode=0 Nov 04 01:31:06.396628 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:06.396457 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr" Nov 04 01:31:06.396628 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:06.396481 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr" event={"ID":"edada0ad-6b72-442b-ade8-b4f02d71f86c","Type":"ContainerDied","Data":"c0cc71a0e0185f4cfacb1f9c7143c79dd959ed165f702f2f9f5cd7d28da9fc37"} Nov 04 01:31:06.396628 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:06.396539 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr" event={"ID":"edada0ad-6b72-442b-ade8-b4f02d71f86c","Type":"ContainerDied","Data":"3100bda9b21be49d7d373abefd4edf003d8c1c1523643ce8e2d9dc10c01e57ef"} Nov 04 01:31:06.396628 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:06.396584 4852 scope.go:122] "RemoveContainer" containerID="c0cc71a0e0185f4cfacb1f9c7143c79dd959ed165f702f2f9f5cd7d28da9fc37" Nov 04 01:31:06.411523 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:06.411470 4852 scope.go:122] "RemoveContainer" containerID="c0cc71a0e0185f4cfacb1f9c7143c79dd959ed165f702f2f9f5cd7d28da9fc37" Nov 04 01:31:06.411999 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:31:06.411970 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0cc71a0e0185f4cfacb1f9c7143c79dd959ed165f702f2f9f5cd7d28da9fc37\": container with ID starting with c0cc71a0e0185f4cfacb1f9c7143c79dd959ed165f702f2f9f5cd7d28da9fc37 not found: ID does not exist" containerID="c0cc71a0e0185f4cfacb1f9c7143c79dd959ed165f702f2f9f5cd7d28da9fc37" Nov 04 01:31:06.412132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:06.412014 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0cc71a0e0185f4cfacb1f9c7143c79dd959ed165f702f2f9f5cd7d28da9fc37"} err="failed to get container status \"c0cc71a0e0185f4cfacb1f9c7143c79dd959ed165f702f2f9f5cd7d28da9fc37\": rpc error: code = NotFound desc = could not find container \"c0cc71a0e0185f4cfacb1f9c7143c79dd959ed165f702f2f9f5cd7d28da9fc37\": container with ID starting with c0cc71a0e0185f4cfacb1f9c7143c79dd959ed165f702f2f9f5cd7d28da9fc37 not found: ID does not exist" Nov 04 01:31:06.433078 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:06.433039 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edada0ad-6b72-442b-ade8-b4f02d71f86c-kube-api-access-ltm66\" (UniqueName: \"kubernetes.io/projected/edada0ad-6b72-442b-ade8-b4f02d71f86c-kube-api-access-ltm66\") pod \"edada0ad-6b72-442b-ade8-b4f02d71f86c\" (UID: \"edada0ad-6b72-442b-ade8-b4f02d71f86c\") " Nov 04 01:31:06.433235 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:06.433158 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edada0ad-6b72-442b-ade8-b4f02d71f86c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edada0ad-6b72-442b-ade8-b4f02d71f86c-plugins-config-volume\") pod \"edada0ad-6b72-442b-ade8-b4f02d71f86c\" (UID: \"edada0ad-6b72-442b-ade8-b4f02d71f86c\") " Nov 04 01:31:06.433868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:06.433710 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edada0ad-6b72-442b-ade8-b4f02d71f86c-plugins-config-volume" pod "edada0ad-6b72-442b-ade8-b4f02d71f86c" (UID: "edada0ad-6b72-442b-ade8-b4f02d71f86c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:06.434348 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:06.434323 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edada0ad-6b72-442b-ade8-b4f02d71f86c-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:06.435930 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:06.435891 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edada0ad-6b72-442b-ade8-b4f02d71f86c-kube-api-access-ltm66" pod "edada0ad-6b72-442b-ade8-b4f02d71f86c" (UID: "edada0ad-6b72-442b-ade8-b4f02d71f86c"). InnerVolumeSpecName "kube-api-access-ltm66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:06.535802 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:06.535767 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltm66\" (UniqueName: \"kubernetes.io/projected/edada0ad-6b72-442b-ade8-b4f02d71f86c-kube-api-access-ltm66\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:06.608298 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:06.608267 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:06.742291 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:06.742254 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr"] Nov 04 01:31:06.785801 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:06.785742 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df6da4c0-5cddbb47f9-rspdr"] Nov 04 01:31:07.183486 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:07.183449 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:07.523341 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:07.523306 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:07.998142 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:07.998065 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edada0ad-6b72-442b-ade8-b4f02d71f86c" path="/var/lib/kubelet/pods/edada0ad-6b72-442b-ade8-b4f02d71f86c/volumes" Nov 04 01:31:08.036067 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:08.036025 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:08.281481 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:08.281444 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:08.794596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:08.792348 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:09.968225 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:09.968163 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:10.768598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:10.768557 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:11.064115 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:11.064000 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:12.003457 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:12.003418 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:12.887591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:12.887544 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:13.348804 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:13.348766 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:13.946606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:13.943614 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:14.370594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:14.370530 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:15.200883 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.196615 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx"] Nov 04 01:31:15.200883 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.197412 4852 cpu_manager.go:414] "containerMap: removing container" podUID="edada0ad-6b72-442b-ade8-b4f02d71f86c" containerName="payload-pre-processing" Nov 04 01:31:15.200883 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.197427 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edada0ad-6b72-442b-ade8-b4f02d71f86c" containerName="payload-pre-processing" Nov 04 01:31:15.200883 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.197607 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="edada0ad-6b72-442b-ade8-b4f02d71f86c" containerName="payload-pre-processing" Nov 04 01:31:15.201786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.201332 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" Nov 04 01:31:15.202234 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.202205 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx"] Nov 04 01:31:15.216893 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.216866 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-50dab0-serving-cert\"" Nov 04 01:31:15.288728 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.288691 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b90a88e-68fc-4841-b289-5af7080408d6-metrics-certs\") pod \"maas-api-e2e-api-b-50dab0-86579ff588-pmkvx\" (UID: \"4b90a88e-68fc-4841-b289-5af7080408d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" Nov 04 01:31:15.288921 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.288818 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnb9z\" (UniqueName: \"kubernetes.io/projected/4b90a88e-68fc-4841-b289-5af7080408d6-kube-api-access-jnb9z\") pod \"maas-api-e2e-api-b-50dab0-86579ff588-pmkvx\" (UID: \"4b90a88e-68fc-4841-b289-5af7080408d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" Nov 04 01:31:15.288921 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.288906 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b90a88e-68fc-4841-b289-5af7080408d6-openshift-service-ca\") pod \"maas-api-e2e-api-b-50dab0-86579ff588-pmkvx\" (UID: \"4b90a88e-68fc-4841-b289-5af7080408d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" Nov 04 01:31:15.289219 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.289195 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b90a88e-68fc-4841-b289-5af7080408d6-maas-api-tls\") pod \"maas-api-e2e-api-b-50dab0-86579ff588-pmkvx\" (UID: \"4b90a88e-68fc-4841-b289-5af7080408d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" Nov 04 01:31:15.330767 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.330730 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:15.397590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.396085 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b90a88e-68fc-4841-b289-5af7080408d6-maas-api-tls\") pod \"maas-api-e2e-api-b-50dab0-86579ff588-pmkvx\" (UID: \"4b90a88e-68fc-4841-b289-5af7080408d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" Nov 04 01:31:15.397590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.396242 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b90a88e-68fc-4841-b289-5af7080408d6-metrics-certs\") pod \"maas-api-e2e-api-b-50dab0-86579ff588-pmkvx\" (UID: \"4b90a88e-68fc-4841-b289-5af7080408d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" Nov 04 01:31:15.397590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.396314 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnb9z\" (UniqueName: \"kubernetes.io/projected/4b90a88e-68fc-4841-b289-5af7080408d6-kube-api-access-jnb9z\") pod \"maas-api-e2e-api-b-50dab0-86579ff588-pmkvx\" (UID: \"4b90a88e-68fc-4841-b289-5af7080408d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" Nov 04 01:31:15.397590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.396407 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b90a88e-68fc-4841-b289-5af7080408d6-openshift-service-ca\") pod \"maas-api-e2e-api-b-50dab0-86579ff588-pmkvx\" (UID: \"4b90a88e-68fc-4841-b289-5af7080408d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" Nov 04 01:31:15.397590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.397497 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b90a88e-68fc-4841-b289-5af7080408d6-openshift-service-ca\") pod \"maas-api-e2e-api-b-50dab0-86579ff588-pmkvx\" (UID: \"4b90a88e-68fc-4841-b289-5af7080408d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" Nov 04 01:31:15.401588 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:31:15.398111 4852 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-serving-cert: secret "maas-api-e2e-api-b-50dab0-serving-cert" not found Nov 04 01:31:15.401588 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:31:15.398181 4852 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b90a88e-68fc-4841-b289-5af7080408d6-maas-api-tls podName:4b90a88e-68fc-4841-b289-5af7080408d6 nodeName:}" failed. No retries permitted until 2026-11-04 01:31:15.898163209 +0000 UTC m=+1958.013273320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b90a88e-68fc-4841-b289-5af7080408d6-maas-api-tls") pod "maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" (UID: "4b90a88e-68fc-4841-b289-5af7080408d6") : secret "maas-api-e2e-api-b-50dab0-serving-cert" not found Nov 04 01:31:15.438593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.437292 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnb9z\" (UniqueName: \"kubernetes.io/projected/4b90a88e-68fc-4841-b289-5af7080408d6-kube-api-access-jnb9z\") pod \"maas-api-e2e-api-b-50dab0-86579ff588-pmkvx\" (UID: \"4b90a88e-68fc-4841-b289-5af7080408d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" Nov 04 01:31:15.438593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.438053 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b90a88e-68fc-4841-b289-5af7080408d6-metrics-certs\") pod \"maas-api-e2e-api-b-50dab0-86579ff588-pmkvx\" (UID: \"4b90a88e-68fc-4841-b289-5af7080408d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" Nov 04 01:31:15.483835 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.483192 4852 generic.go:356] "Generic (PLEG): container finished" podID="207f3c0f-a068-4aee-a593-88e94bcc0dff" containerID="28dc1bfb2ceae7411142f3726755faf4c03092336b4000df1e7d2683547abe2d" exitCode=0 Nov 04 01:31:15.483835 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.483338 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" event={"ID":"207f3c0f-a068-4aee-a593-88e94bcc0dff","Type":"ContainerDied","Data":"28dc1bfb2ceae7411142f3726755faf4c03092336b4000df1e7d2683547abe2d"} Nov 04 01:31:15.484031 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.483909 4852 scope.go:122] "RemoveContainer" containerID="28dc1bfb2ceae7411142f3726755faf4c03092336b4000df1e7d2683547abe2d" Nov 04 01:31:15.486604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.486506 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:31:15.486726 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.486649 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:31:15.486867 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.486841 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:31:15.559542 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.559503 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:15.791588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.790643 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:15.918599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.915780 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b90a88e-68fc-4841-b289-5af7080408d6-maas-api-tls\") pod \"maas-api-e2e-api-b-50dab0-86579ff588-pmkvx\" (UID: \"4b90a88e-68fc-4841-b289-5af7080408d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" Nov 04 01:31:15.944606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:15.941846 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b90a88e-68fc-4841-b289-5af7080408d6-maas-api-tls\") pod \"maas-api-e2e-api-b-50dab0-86579ff588-pmkvx\" (UID: \"4b90a88e-68fc-4841-b289-5af7080408d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" Nov 04 01:31:16.130126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:16.130039 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" Nov 04 01:31:16.470413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:16.470281 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:16.821819 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:16.815225 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" event={"ID":"207f3c0f-a068-4aee-a593-88e94bcc0dff","Type":"ContainerStarted","Data":"9885f7bd471fdded0f006a03ede0c9f0092a4c5fd9784620552b3ac8e8a677f7"} Nov 04 01:31:17.099078 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:17.099046 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx"] Nov 04 01:31:17.099387 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:31:17.099359 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b90a88e_68fc_4841_b289_5af7080408d6.slice/crio-53c18586de56a2eb718b913c22ec6e16d14edc5ff4ab2cfedd1facbe4e00e3a1 WatchSource:0}: Error finding container 53c18586de56a2eb718b913c22ec6e16d14edc5ff4ab2cfedd1facbe4e00e3a1: Status 404 returned error can't find the container with id 53c18586de56a2eb718b913c22ec6e16d14edc5ff4ab2cfedd1facbe4e00e3a1 Nov 04 01:31:17.101351 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:17.101334 4852 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:31:17.242646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:17.242604 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:17.477651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:17.472188 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:17.519819 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:17.519786 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:31:17.520927 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:17.520898 4852 patch_prober.go:29] interesting pod/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:31:17.521061 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:17.520947 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" podUID="207f3c0f-a068-4aee-a593-88e94bcc0dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 04 01:31:17.701484 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:17.701385 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:17.883602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:17.883206 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" event={"ID":"4b90a88e-68fc-4841-b289-5af7080408d6","Type":"ContainerStarted","Data":"b2fe84c03406329972b52b0a5d64da509f76e6126f1875cf62e94004b4efad49"} Nov 04 01:31:17.883602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:17.883265 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" event={"ID":"4b90a88e-68fc-4841-b289-5af7080408d6","Type":"ContainerStarted","Data":"53c18586de56a2eb718b913c22ec6e16d14edc5ff4ab2cfedd1facbe4e00e3a1"} Nov 04 01:31:17.883602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:17.883293 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" Nov 04 01:31:17.907587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:17.903985 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:18.533624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:18.532173 4852 patch_prober.go:29] interesting pod/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:31:18.533624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:18.532232 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" podUID="207f3c0f-a068-4aee-a593-88e94bcc0dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 04 01:31:18.714784 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:18.713891 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:18.992638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:18.987490 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:19.254732 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:19.236154 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:19.595640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:19.565744 4852 patch_prober.go:29] interesting pod/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:31:19.595640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:19.565819 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" podUID="207f3c0f-a068-4aee-a593-88e94bcc0dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 04 01:31:19.797630 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:19.794760 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:19.899411 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:19.899317 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:31:19.990597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:19.986704 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:20.520226 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:20.519804 4852 patch_prober.go:29] interesting pod/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:31:20.520226 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:20.519867 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" podUID="207f3c0f-a068-4aee-a593-88e94bcc0dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 04 01:31:20.655816 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:20.655779 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:21.446842 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:21.428546 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:21.527618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:21.527030 4852 patch_prober.go:29] interesting pod/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:31:21.527618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:21.527094 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" podUID="207f3c0f-a068-4aee-a593-88e94bcc0dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 04 01:31:22.047304 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:22.044086 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:22.237858 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:22.237817 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:22.387396 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:22.387304 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:22.520308 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:22.520266 4852 patch_prober.go:29] interesting pod/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:31:22.520488 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:22.520324 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" podUID="207f3c0f-a068-4aee-a593-88e94bcc0dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 04 01:31:22.959596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:22.959091 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:23.410959 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:23.410924 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:23.559588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:23.547025 4852 patch_prober.go:29] interesting pod/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:31:23.573588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:23.571636 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" podUID="207f3c0f-a068-4aee-a593-88e94bcc0dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 04 01:31:23.642484 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:23.642445 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:24.147586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:24.144079 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:24.255593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:24.255405 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" Nov 04 01:31:24.284292 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:24.284225 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" podStartSLOduration=8.794171254 podStartE2EDuration="9.284210013s" totalImagesPullingTime="490.038759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:17.958528529 +0000 UTC m=+1960.073638640" watchObservedRunningTime="2026-11-04 01:31:24.284210013 +0000 UTC m=+1966.399320132" Nov 04 01:31:24.520359 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:24.520325 4852 patch_prober.go:29] interesting pod/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:31:24.520858 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:24.520379 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" podUID="207f3c0f-a068-4aee-a593-88e94bcc0dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 04 01:31:24.960359 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:24.960267 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:25.525518 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:25.525490 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:31:25.526853 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:25.526822 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:31:25.744970 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:25.744914 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:26.518202 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:26.518149 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:26.930881 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:26.930756 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:27.897592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:27.897436 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:29.056610 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:29.015632 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:29.943054 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:31:29.943014 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:31:30.096592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:30.095878 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:30.560595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:30.560008 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:30.759942 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:30.759903 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:30.966996 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:30.966907 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:31.350914 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:31.350878 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:31.526954 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:31.526922 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524"] Nov 04 01:31:31.527192 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:31.527165 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" podUID="158b1659-369b-454e-bb71-db3763ca6774" containerName="istio-proxy" containerID="cri-o://666b7511555868d7af6e2fcefbb8cecb2463ae9df141e161879443c14cf4278a" gracePeriod=30 Nov 04 01:31:31.544252 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:31.544222 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb"] Nov 04 01:31:31.544435 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:31.544416 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" podUID="85f24a4b-278a-432f-a86b-f3addc378bee" containerName="istio-proxy" containerID="cri-o://4662529fdebde9b66f372fcfbce7255f8677c0f42baa49b09de1c50d0217ebce" gracePeriod=30 Nov 04 01:31:31.568888 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:31.568852 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8"] Nov 04 01:31:31.569148 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:31.569116 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" podUID="107a1fd6-feb8-4615-91ef-980f9af78d5f" containerName="istio-proxy" containerID="cri-o://da3bc239519edadea7f880ceaaef53b2fa50c586d65aff4fcada07143be32366" gracePeriod=30 Nov 04 01:31:31.657148 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:31.657059 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:32.419365 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:32.419326 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:32.584273 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:32.584237 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:32.975068 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:32.975014 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:33.204697 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:33.194389 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:33.575365 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:33.575269 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:34.323057 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:34.323010 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:35.322393 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:35.322360 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:35.540767 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:35.540729 4852 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:31:35.541095 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:35.541057 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" podUID="107a1fd6-feb8-4615-91ef-980f9af78d5f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:31:35.563133 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:35.563097 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:36.140598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:36.139475 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:36.518012 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:36.517968 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:36.734199 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:36.733917 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:37.241191 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.241015 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:37.261482 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.261440 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:31:37.300153 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.300086 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:31:37.338641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.338553 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:31:37.354428 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.354367 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/158b1659-369b-454e-bb71-db3763ca6774-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158b1659-369b-454e-bb71-db3763ca6774-istiod-ca-cert\") pod \"158b1659-369b-454e-bb71-db3763ca6774\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " Nov 04 01:31:37.354678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.354656 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-workload-certs\") pod \"158b1659-369b-454e-bb71-db3763ca6774\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " Nov 04 01:31:37.354773 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.354756 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-credential-socket\") pod \"158b1659-369b-454e-bb71-db3763ca6774\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " Nov 04 01:31:37.354872 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.354857 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-workload-socket\") pod \"158b1659-369b-454e-bb71-db3763ca6774\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " Nov 04 01:31:37.355159 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.355139 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-istio-envoy\") pod \"158b1659-369b-454e-bb71-db3763ca6774\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " Nov 04 01:31:37.355258 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.355217 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158b1659-369b-454e-bb71-db3763ca6774-kube-api-access-gpqdj\" (UniqueName: \"kubernetes.io/projected/158b1659-369b-454e-bb71-db3763ca6774-kube-api-access-gpqdj\") pod \"158b1659-369b-454e-bb71-db3763ca6774\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " Nov 04 01:31:37.355315 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.355259 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158b1659-369b-454e-bb71-db3763ca6774-istio-token\" (UniqueName: \"kubernetes.io/projected/158b1659-369b-454e-bb71-db3763ca6774-istio-token\") pod \"158b1659-369b-454e-bb71-db3763ca6774\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " Nov 04 01:31:37.355370 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.355335 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/158b1659-369b-454e-bb71-db3763ca6774-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158b1659-369b-454e-bb71-db3763ca6774-istio-podinfo\") pod \"158b1659-369b-454e-bb71-db3763ca6774\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " Nov 04 01:31:37.355432 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.355378 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-istio-data\") pod \"158b1659-369b-454e-bb71-db3763ca6774\" (UID: \"158b1659-369b-454e-bb71-db3763ca6774\") " Nov 04 01:31:37.356470 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.356442 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-istio-data" pod "158b1659-369b-454e-bb71-db3763ca6774" (UID: "158b1659-369b-454e-bb71-db3763ca6774"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.356876 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.356854 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158b1659-369b-454e-bb71-db3763ca6774-istiod-ca-cert" pod "158b1659-369b-454e-bb71-db3763ca6774" (UID: "158b1659-369b-454e-bb71-db3763ca6774"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:37.357223 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.357200 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-workload-certs" pod "158b1659-369b-454e-bb71-db3763ca6774" (UID: "158b1659-369b-454e-bb71-db3763ca6774"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.357409 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.357389 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-credential-socket" pod "158b1659-369b-454e-bb71-db3763ca6774" (UID: "158b1659-369b-454e-bb71-db3763ca6774"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.358346 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.358320 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-workload-socket" pod "158b1659-369b-454e-bb71-db3763ca6774" (UID: "158b1659-369b-454e-bb71-db3763ca6774"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.361611 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.360460 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158b1659-369b-454e-bb71-db3763ca6774-kube-api-access-gpqdj" pod "158b1659-369b-454e-bb71-db3763ca6774" (UID: "158b1659-369b-454e-bb71-db3763ca6774"). InnerVolumeSpecName "kube-api-access-gpqdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:37.361611 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.360618 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/158b1659-369b-454e-bb71-db3763ca6774-istio-podinfo" pod "158b1659-369b-454e-bb71-db3763ca6774" (UID: "158b1659-369b-454e-bb71-db3763ca6774"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:31:37.362547 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.362528 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158b1659-369b-454e-bb71-db3763ca6774-istio-token" pod "158b1659-369b-454e-bb71-db3763ca6774" (UID: "158b1659-369b-454e-bb71-db3763ca6774"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:37.362691 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.362632 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-istio-envoy" pod "158b1659-369b-454e-bb71-db3763ca6774" (UID: "158b1659-369b-454e-bb71-db3763ca6774"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.457996 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.457834 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-istio-envoy\") pod \"85f24a4b-278a-432f-a86b-f3addc378bee\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " Nov 04 01:31:37.457996 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.457917 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-credential-socket\") pod \"85f24a4b-278a-432f-a86b-f3addc378bee\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " Nov 04 01:31:37.458785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.458044 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-envoy\") pod \"107a1fd6-feb8-4615-91ef-980f9af78d5f\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " Nov 04 01:31:37.458785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.458121 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-podinfo\") pod \"107a1fd6-feb8-4615-91ef-980f9af78d5f\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " Nov 04 01:31:37.458785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.458183 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85f24a4b-278a-432f-a86b-f3addc378bee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f24a4b-278a-432f-a86b-f3addc378bee-istio-podinfo\") pod \"85f24a4b-278a-432f-a86b-f3addc378bee\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " Nov 04 01:31:37.458785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.458249 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-istio-data\") pod \"85f24a4b-278a-432f-a86b-f3addc378bee\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " Nov 04 01:31:37.458785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.458281 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f24a4b-278a-432f-a86b-f3addc378bee-istio-token\" (UniqueName: \"kubernetes.io/projected/85f24a4b-278a-432f-a86b-f3addc378bee-istio-token\") pod \"85f24a4b-278a-432f-a86b-f3addc378bee\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " Nov 04 01:31:37.458785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.458314 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f24a4b-278a-432f-a86b-f3addc378bee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f24a4b-278a-432f-a86b-f3addc378bee-istiod-ca-cert\") pod \"85f24a4b-278a-432f-a86b-f3addc378bee\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " Nov 04 01:31:37.458785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.458376 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-credential-socket\") pod \"107a1fd6-feb8-4615-91ef-980f9af78d5f\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " Nov 04 01:31:37.458785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.458421 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-workload-certs\") pod \"107a1fd6-feb8-4615-91ef-980f9af78d5f\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " Nov 04 01:31:37.458785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.458447 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-token\" (UniqueName: \"kubernetes.io/projected/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-token\") pod \"107a1fd6-feb8-4615-91ef-980f9af78d5f\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " Nov 04 01:31:37.458785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.458503 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f24a4b-278a-432f-a86b-f3addc378bee-kube-api-access-xrjlw\" (UniqueName: \"kubernetes.io/projected/85f24a4b-278a-432f-a86b-f3addc378bee-kube-api-access-xrjlw\") pod \"85f24a4b-278a-432f-a86b-f3addc378bee\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " Nov 04 01:31:37.458785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.458547 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-data\") pod \"107a1fd6-feb8-4615-91ef-980f9af78d5f\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " Nov 04 01:31:37.458785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.458546 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-credential-socket" pod "85f24a4b-278a-432f-a86b-f3addc378bee" (UID: "85f24a4b-278a-432f-a86b-f3addc378bee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.458785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.458628 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-workload-socket\") pod \"107a1fd6-feb8-4615-91ef-980f9af78d5f\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " Nov 04 01:31:37.458785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.458738 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-workload-certs\") pod \"85f24a4b-278a-432f-a86b-f3addc378bee\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " Nov 04 01:31:37.458785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.458771 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-workload-socket\") pod \"85f24a4b-278a-432f-a86b-f3addc378bee\" (UID: \"85f24a4b-278a-432f-a86b-f3addc378bee\") " Nov 04 01:31:37.459759 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.458861 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/107a1fd6-feb8-4615-91ef-980f9af78d5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107a1fd6-feb8-4615-91ef-980f9af78d5f-istiod-ca-cert\") pod \"107a1fd6-feb8-4615-91ef-980f9af78d5f\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " Nov 04 01:31:37.459759 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.458916 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107a1fd6-feb8-4615-91ef-980f9af78d5f-kube-api-access-mtdwf\" (UniqueName: \"kubernetes.io/projected/107a1fd6-feb8-4615-91ef-980f9af78d5f-kube-api-access-mtdwf\") pod \"107a1fd6-feb8-4615-91ef-980f9af78d5f\" (UID: \"107a1fd6-feb8-4615-91ef-980f9af78d5f\") " Nov 04 01:31:37.459759 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.459004 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f24a4b-278a-432f-a86b-f3addc378bee-istiod-ca-cert" pod "85f24a4b-278a-432f-a86b-f3addc378bee" (UID: "85f24a4b-278a-432f-a86b-f3addc378bee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:37.459920 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.459871 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.459920 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.459893 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.459920 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.459909 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f24a4b-278a-432f-a86b-f3addc378bee-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.460079 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.459923 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.460079 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.459936 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpqdj\" (UniqueName: \"kubernetes.io/projected/158b1659-369b-454e-bb71-db3763ca6774-kube-api-access-gpqdj\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.460079 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.459950 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158b1659-369b-454e-bb71-db3763ca6774-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.460079 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.459964 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158b1659-369b-454e-bb71-db3763ca6774-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.460079 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.459979 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.460079 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.459995 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.460079 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.460009 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158b1659-369b-454e-bb71-db3763ca6774-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.460079 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.460022 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158b1659-369b-454e-bb71-db3763ca6774-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.460505 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.460407 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-data" pod "107a1fd6-feb8-4615-91ef-980f9af78d5f" (UID: "107a1fd6-feb8-4615-91ef-980f9af78d5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.460647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.460621 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-credential-socket" pod "107a1fd6-feb8-4615-91ef-980f9af78d5f" (UID: "107a1fd6-feb8-4615-91ef-980f9af78d5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.460832 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.460810 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-workload-certs" pod "107a1fd6-feb8-4615-91ef-980f9af78d5f" (UID: "107a1fd6-feb8-4615-91ef-980f9af78d5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.461554 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.461526 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-istio-data" pod "85f24a4b-278a-432f-a86b-f3addc378bee" (UID: "85f24a4b-278a-432f-a86b-f3addc378bee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.462006 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.461978 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-envoy" pod "107a1fd6-feb8-4615-91ef-980f9af78d5f" (UID: "107a1fd6-feb8-4615-91ef-980f9af78d5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.462312 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.462288 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-workload-certs" pod "85f24a4b-278a-432f-a86b-f3addc378bee" (UID: "85f24a4b-278a-432f-a86b-f3addc378bee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.462499 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.462477 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-workload-socket" pod "107a1fd6-feb8-4615-91ef-980f9af78d5f" (UID: "107a1fd6-feb8-4615-91ef-980f9af78d5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.462713 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.462691 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-workload-socket" pod "85f24a4b-278a-432f-a86b-f3addc378bee" (UID: "85f24a4b-278a-432f-a86b-f3addc378bee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.462797 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.462771 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107a1fd6-feb8-4615-91ef-980f9af78d5f-kube-api-access-mtdwf" pod "107a1fd6-feb8-4615-91ef-980f9af78d5f" (UID: "107a1fd6-feb8-4615-91ef-980f9af78d5f"). InnerVolumeSpecName "kube-api-access-mtdwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:37.463026 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.463004 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107a1fd6-feb8-4615-91ef-980f9af78d5f-istiod-ca-cert" pod "107a1fd6-feb8-4615-91ef-980f9af78d5f" (UID: "107a1fd6-feb8-4615-91ef-980f9af78d5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:37.463102 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.463040 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-istio-envoy" pod "85f24a4b-278a-432f-a86b-f3addc378bee" (UID: "85f24a4b-278a-432f-a86b-f3addc378bee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.464637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.464595 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-podinfo" pod "107a1fd6-feb8-4615-91ef-980f9af78d5f" (UID: "107a1fd6-feb8-4615-91ef-980f9af78d5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:31:37.464748 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.464652 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85f24a4b-278a-432f-a86b-f3addc378bee-istio-podinfo" pod "85f24a4b-278a-432f-a86b-f3addc378bee" (UID: "85f24a4b-278a-432f-a86b-f3addc378bee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:31:37.464815 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.464763 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f24a4b-278a-432f-a86b-f3addc378bee-istio-token" pod "85f24a4b-278a-432f-a86b-f3addc378bee" (UID: "85f24a4b-278a-432f-a86b-f3addc378bee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:37.465818 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.465787 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-token" pod "107a1fd6-feb8-4615-91ef-980f9af78d5f" (UID: "107a1fd6-feb8-4615-91ef-980f9af78d5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:37.466928 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.466894 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f24a4b-278a-432f-a86b-f3addc378bee-kube-api-access-xrjlw" pod "85f24a4b-278a-432f-a86b-f3addc378bee" (UID: "85f24a4b-278a-432f-a86b-f3addc378bee"). InnerVolumeSpecName "kube-api-access-xrjlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:37.561710 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.561679 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.561710 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.561711 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.562135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.561724 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107a1fd6-feb8-4615-91ef-980f9af78d5f-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.562135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.561737 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtdwf\" (UniqueName: \"kubernetes.io/projected/107a1fd6-feb8-4615-91ef-980f9af78d5f-kube-api-access-mtdwf\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.562135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.561750 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.562135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.561764 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.562135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.561777 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.562135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.561803 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f24a4b-278a-432f-a86b-f3addc378bee-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.562135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.561817 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f24a4b-278a-432f-a86b-f3addc378bee-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.562135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.561831 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f24a4b-278a-432f-a86b-f3addc378bee-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.562135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.561847 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.562135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.561861 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.562135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.561876 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.562135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.561889 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrjlw\" (UniqueName: \"kubernetes.io/projected/85f24a4b-278a-432f-a86b-f3addc378bee-kube-api-access-xrjlw\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.562135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.561903 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.562135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.561916 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107a1fd6-feb8-4615-91ef-980f9af78d5f-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.602652 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.602605 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:37.630311 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.630279 4852 generic.go:356] "Generic (PLEG): container finished" podID="107a1fd6-feb8-4615-91ef-980f9af78d5f" containerID="da3bc239519edadea7f880ceaaef53b2fa50c586d65aff4fcada07143be32366" exitCode=0 Nov 04 01:31:37.630502 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.630340 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" Nov 04 01:31:37.630502 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.630362 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" event={"ID":"107a1fd6-feb8-4615-91ef-980f9af78d5f","Type":"ContainerDied","Data":"da3bc239519edadea7f880ceaaef53b2fa50c586d65aff4fcada07143be32366"} Nov 04 01:31:37.630502 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.630423 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8" event={"ID":"107a1fd6-feb8-4615-91ef-980f9af78d5f","Type":"ContainerDied","Data":"fd39a7e86826d2f223c18ab7a2b5a9c80d433d3b0ab3d969853803231bb26488"} Nov 04 01:31:37.630502 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.630455 4852 scope.go:122] "RemoveContainer" containerID="da3bc239519edadea7f880ceaaef53b2fa50c586d65aff4fcada07143be32366" Nov 04 01:31:37.631949 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.631926 4852 generic.go:356] "Generic (PLEG): container finished" podID="158b1659-369b-454e-bb71-db3763ca6774" containerID="666b7511555868d7af6e2fcefbb8cecb2463ae9df141e161879443c14cf4278a" exitCode=0 Nov 04 01:31:37.632051 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.631974 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" Nov 04 01:31:37.632051 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.631985 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" event={"ID":"158b1659-369b-454e-bb71-db3763ca6774","Type":"ContainerDied","Data":"666b7511555868d7af6e2fcefbb8cecb2463ae9df141e161879443c14cf4278a"} Nov 04 01:31:37.632051 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.632035 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524" event={"ID":"158b1659-369b-454e-bb71-db3763ca6774","Type":"ContainerDied","Data":"2d626a7eea6878f3c75ec0c6c6ef97b7d6ea8625af9f77a234beaf7e7b707ed7"} Nov 04 01:31:37.633919 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.633886 4852 generic.go:356] "Generic (PLEG): container finished" podID="85f24a4b-278a-432f-a86b-f3addc378bee" containerID="4662529fdebde9b66f372fcfbce7255f8677c0f42baa49b09de1c50d0217ebce" exitCode=0 Nov 04 01:31:37.634054 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.633942 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" Nov 04 01:31:37.634188 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.634148 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" event={"ID":"85f24a4b-278a-432f-a86b-f3addc378bee","Type":"ContainerDied","Data":"4662529fdebde9b66f372fcfbce7255f8677c0f42baa49b09de1c50d0217ebce"} Nov 04 01:31:37.634326 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.634226 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb" event={"ID":"85f24a4b-278a-432f-a86b-f3addc378bee","Type":"ContainerDied","Data":"e06b4a783ba3ee22e9825f8c2988a63417b395cf486b3d9865cf8aff4d57a4b7"} Nov 04 01:31:37.643273 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.643186 4852 scope.go:122] "RemoveContainer" containerID="da3bc239519edadea7f880ceaaef53b2fa50c586d65aff4fcada07143be32366" Nov 04 01:31:37.643664 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:31:37.643599 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da3bc239519edadea7f880ceaaef53b2fa50c586d65aff4fcada07143be32366\": container with ID starting with da3bc239519edadea7f880ceaaef53b2fa50c586d65aff4fcada07143be32366 not found: ID does not exist" containerID="da3bc239519edadea7f880ceaaef53b2fa50c586d65aff4fcada07143be32366" Nov 04 01:31:37.643664 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.643627 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da3bc239519edadea7f880ceaaef53b2fa50c586d65aff4fcada07143be32366"} err="failed to get container status \"da3bc239519edadea7f880ceaaef53b2fa50c586d65aff4fcada07143be32366\": rpc error: code = NotFound desc = could not find container \"da3bc239519edadea7f880ceaaef53b2fa50c586d65aff4fcada07143be32366\": container with ID starting with da3bc239519edadea7f880ceaaef53b2fa50c586d65aff4fcada07143be32366 not found: ID does not exist" Nov 04 01:31:37.643664 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.643646 4852 scope.go:122] "RemoveContainer" containerID="666b7511555868d7af6e2fcefbb8cecb2463ae9df141e161879443c14cf4278a" Nov 04 01:31:37.656493 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.656465 4852 scope.go:122] "RemoveContainer" containerID="666b7511555868d7af6e2fcefbb8cecb2463ae9df141e161879443c14cf4278a" Nov 04 01:31:37.656896 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:31:37.656872 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"666b7511555868d7af6e2fcefbb8cecb2463ae9df141e161879443c14cf4278a\": container with ID starting with 666b7511555868d7af6e2fcefbb8cecb2463ae9df141e161879443c14cf4278a not found: ID does not exist" containerID="666b7511555868d7af6e2fcefbb8cecb2463ae9df141e161879443c14cf4278a" Nov 04 01:31:37.657011 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.656902 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"666b7511555868d7af6e2fcefbb8cecb2463ae9df141e161879443c14cf4278a"} err="failed to get container status \"666b7511555868d7af6e2fcefbb8cecb2463ae9df141e161879443c14cf4278a\": rpc error: code = NotFound desc = could not find container \"666b7511555868d7af6e2fcefbb8cecb2463ae9df141e161879443c14cf4278a\": container with ID starting with 666b7511555868d7af6e2fcefbb8cecb2463ae9df141e161879443c14cf4278a not found: ID does not exist" Nov 04 01:31:37.657011 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.656919 4852 scope.go:122] "RemoveContainer" containerID="4662529fdebde9b66f372fcfbce7255f8677c0f42baa49b09de1c50d0217ebce" Nov 04 01:31:37.662837 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.662811 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8"] Nov 04 01:31:37.669131 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.669109 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-r8lm8"] Nov 04 01:31:37.670018 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.669991 4852 scope.go:122] "RemoveContainer" containerID="4662529fdebde9b66f372fcfbce7255f8677c0f42baa49b09de1c50d0217ebce" Nov 04 01:31:37.670307 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:31:37.670290 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4662529fdebde9b66f372fcfbce7255f8677c0f42baa49b09de1c50d0217ebce\": container with ID starting with 4662529fdebde9b66f372fcfbce7255f8677c0f42baa49b09de1c50d0217ebce not found: ID does not exist" containerID="4662529fdebde9b66f372fcfbce7255f8677c0f42baa49b09de1c50d0217ebce" Nov 04 01:31:37.670379 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.670322 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4662529fdebde9b66f372fcfbce7255f8677c0f42baa49b09de1c50d0217ebce"} err="failed to get container status \"4662529fdebde9b66f372fcfbce7255f8677c0f42baa49b09de1c50d0217ebce\": rpc error: code = NotFound desc = could not find container \"4662529fdebde9b66f372fcfbce7255f8677c0f42baa49b09de1c50d0217ebce\": container with ID starting with 4662529fdebde9b66f372fcfbce7255f8677c0f42baa49b09de1c50d0217ebce not found: ID does not exist" Nov 04 01:31:37.682864 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.682841 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524"] Nov 04 01:31:37.690533 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.690509 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kh524"] Nov 04 01:31:37.703618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.703551 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb"] Nov 04 01:31:37.709106 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.709037 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ppdrb"] Nov 04 01:31:37.772034 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.771997 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:37.951106 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:37.951068 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:38.039921 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:38.039890 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="107a1fd6-feb8-4615-91ef-980f9af78d5f" path="/var/lib/kubelet/pods/107a1fd6-feb8-4615-91ef-980f9af78d5f/volumes" Nov 04 01:31:38.040517 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:38.040502 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="158b1659-369b-454e-bb71-db3763ca6774" path="/var/lib/kubelet/pods/158b1659-369b-454e-bb71-db3763ca6774/volumes" Nov 04 01:31:38.041079 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:38.041063 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f24a4b-278a-432f-a86b-f3addc378bee" path="/var/lib/kubelet/pods/85f24a4b-278a-432f-a86b-f3addc378bee/volumes" Nov 04 01:31:38.143280 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:38.143252 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:38.439201 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:38.439111 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:38.783644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:38.720682 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:39.042868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:39.040646 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:39.764586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:39.761230 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:40.436732 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:40.436701 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:40.596442 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:40.596370 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:41.332233 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:41.332198 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:42.198076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:42.197988 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:42.544955 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:42.540087 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:42.997291 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:42.997155 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:44.008108 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:44.008069 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:44.637824 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:44.637252 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:45.086119 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:45.086068 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:45.250422 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:45.250385 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:45.457650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:45.435292 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:45.832206 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:45.832168 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:46.734998 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:46.734963 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:47.429460 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:47.429422 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz"] Nov 04 01:31:47.527231 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:47.527192 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:47.741281 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:47.741188 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:48.375328 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:48.375261 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:49.286984 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:49.286953 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:50.214041 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:50.214006 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:51.076138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:51.076102 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:51.827554 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:51.827513 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:52.044626 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:52.044597 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:52.752282 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:52.752234 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:53.545451 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:53.545413 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:54.628585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:54.626888 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:54.813911 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:54.813876 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:54.988521 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:54.988427 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:55.892411 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:55.892378 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:56.691438 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:56.691370 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:57.605454 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:57.605403 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:58.014237 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:58.010405 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:58.471631 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:58.471509 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:58.676526 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:58.676494 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:59.457602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:59.423858 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:59.637591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:59.633755 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:31:59.911270 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:31:59.911233 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:31:59.932373 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:31:59.932336 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:00.464104 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:00.464070 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:01.159336 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:01.159300 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:32:01.714098 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:01.714036 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:02.452930 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:02.452885 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" podUID="b1aea2bc-8d86-4c48-8ce7-e0182f02328b" containerName="main" containerID="cri-o://d6d46115bafd6a2bae1a5dd623d90c4af5a38fedf1bd43c6c4a1e487176147ed" gracePeriod=45 Nov 04 01:32:02.620759 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:02.620722 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:02.730532 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:02.730427 4852 generic.go:356] "Generic (PLEG): container finished" podID="b1aea2bc-8d86-4c48-8ce7-e0182f02328b" containerID="d6d46115bafd6a2bae1a5dd623d90c4af5a38fedf1bd43c6c4a1e487176147ed" exitCode=0 Nov 04 01:32:02.731044 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:02.730791 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" event={"ID":"b1aea2bc-8d86-4c48-8ce7-e0182f02328b","Type":"ContainerDied","Data":"d6d46115bafd6a2bae1a5dd623d90c4af5a38fedf1bd43c6c4a1e487176147ed"} Nov 04 01:32:02.826144 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:02.826048 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:03.033834 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.033787 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:32:03.203893 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.192666 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-kube-api-access-s8nfr\" (UniqueName: \"kubernetes.io/projected/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-kube-api-access-s8nfr\") pod \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " Nov 04 01:32:03.203893 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.192834 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-home\" (UniqueName: \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-home\") pod \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " Nov 04 01:32:03.203893 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.192896 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-tmp-dir\") pod \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " Nov 04 01:32:03.203893 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.192991 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-dshm\") pod \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " Nov 04 01:32:03.203893 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.193018 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-model-cache\") pod \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " Nov 04 01:32:03.203893 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.193198 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-tls-certs\") pod \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\" (UID: \"b1aea2bc-8d86-4c48-8ce7-e0182f02328b\") " Nov 04 01:32:03.203893 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.203016 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-model-cache" pod "b1aea2bc-8d86-4c48-8ce7-e0182f02328b" (UID: "b1aea2bc-8d86-4c48-8ce7-e0182f02328b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:03.203893 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.203338 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-tmp-dir" pod "b1aea2bc-8d86-4c48-8ce7-e0182f02328b" (UID: "b1aea2bc-8d86-4c48-8ce7-e0182f02328b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:03.205676 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.205637 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-home" pod "b1aea2bc-8d86-4c48-8ce7-e0182f02328b" (UID: "b1aea2bc-8d86-4c48-8ce7-e0182f02328b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:03.357637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.351843 4852 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-home\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:03.357637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.351883 4852 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-tmp-dir\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:03.357637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.351896 4852 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-model-cache\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:03.377696 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.372031 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-kube-api-access-s8nfr" pod "b1aea2bc-8d86-4c48-8ce7-e0182f02328b" (UID: "b1aea2bc-8d86-4c48-8ce7-e0182f02328b"). InnerVolumeSpecName "kube-api-access-s8nfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:03.394113 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.393763 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-tls-certs" pod "b1aea2bc-8d86-4c48-8ce7-e0182f02328b" (UID: "b1aea2bc-8d86-4c48-8ce7-e0182f02328b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:03.394449 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.394408 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-dshm" pod "b1aea2bc-8d86-4c48-8ce7-e0182f02328b" (UID: "b1aea2bc-8d86-4c48-8ce7-e0182f02328b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:03.451601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.415848 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:03.473941 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.473906 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8nfr\" (UniqueName: \"kubernetes.io/projected/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-kube-api-access-s8nfr\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:03.474491 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.474470 4852 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-dshm\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:03.474617 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.474605 4852 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1aea2bc-8d86-4c48-8ce7-e0182f02328b-tls-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:03.746662 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.710203 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" podUID="b1aea2bc-8d86-4c48-8ce7-e0182f02328b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:03.869589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.828209 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:03.889589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.880936 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" event={"ID":"b1aea2bc-8d86-4c48-8ce7-e0182f02328b","Type":"ContainerDied","Data":"3c40d6f2dbe1c70c19a7a727da90ba100b1b662edb0eb556f5cdb918b64362fc"} Nov 04 01:32:03.889589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.881004 4852 scope.go:122] "RemoveContainer" containerID="d6d46115bafd6a2bae1a5dd623d90c4af5a38fedf1bd43c6c4a1e487176147ed" Nov 04 01:32:03.889589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:03.881210 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz" Nov 04 01:32:04.188121 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:04.187133 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 04 01:32:04.188121 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:04.187547 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": context deadline exceeded" Nov 04 01:32:04.228833 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:04.228790 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:04.247773 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:04.237648 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz"] Nov 04 01:32:04.255583 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:04.252641 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3888c434/e2e-lifecycle-model-3888c434-kserve-6cf8bbf5df-47lpz"] Nov 04 01:32:04.366673 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:04.361808 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:04.377609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:04.372644 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 04 01:32:04.377609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:04.373753 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 04 01:32:04.668579 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:04.668447 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:04.670181 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:04.670147 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:04.736159 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:04.724113 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 04 01:32:04.754734 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:04.754689 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:05.001985 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:04.932979 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:05.526805 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:05.513908 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 04 01:32:05.552523 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:05.552495 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:32:05.555657 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:05.543239 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:05.865116 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:05.865028 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:05.989400 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:05.989370 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:32:06.063775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:06.063744 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1aea2bc-8d86-4c48-8ce7-e0182f02328b" path="/var/lib/kubelet/pods/b1aea2bc-8d86-4c48-8ce7-e0182f02328b/volumes" Nov 04 01:32:06.198341 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:06.198206 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:06.353035 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:06.352993 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:06.589839 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:06.587787 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:06.943647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:06.936993 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:07.348590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:07.344654 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:07.363587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:07.359942 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:07.363587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:07.360464 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:32:07.640653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:07.634507 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:32:07.640653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:07.636363 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:32:07.686586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:07.678701 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:07.952388 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:07.952297 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:32:08.097934 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:08.097888 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:08.347879 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:08.332917 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:08.541035 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:08.541000 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:32:08.994631 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:08.993397 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:09.364599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:09.364304 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:32:09.473616 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:09.384742 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:09.473616 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:09.385184 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 04 01:32:09.473616 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:09.386948 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:09.539609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:09.536354 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:09.583271 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:09.583218 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 04 01:32:09.740528 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:09.740424 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:09.903590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:09.886674 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": context deadline exceeded" Nov 04 01:32:10.002647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:10.000019 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 04 01:32:10.002647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:10.000614 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:32:10.054010 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:10.044733 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:10.134620 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:10.131794 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:32:10.288665 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:10.275399 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:10.434592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:10.383822 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 04 01:32:10.471974 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:10.471946 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:32:10.579871 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:10.575030 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:32:11.106401 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:11.106363 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:11.777364 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:11.775420 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:12.028644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:12.025447 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:12.390657 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:12.389690 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:12.657650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:12.657546 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:12.771838 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:12.771806 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7"] Nov 04 01:32:12.772095 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:12.772069 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" podUID="207f3c0f-a068-4aee-a593-88e94bcc0dff" containerName="istio-proxy" containerID="cri-o://9885f7bd471fdded0f006a03ede0c9f0092a4c5fd9784620552b3ac8e8a677f7" gracePeriod=30 Nov 04 01:32:12.917105 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:12.917015 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:13.573675 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:13.573548 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:13.845801 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:13.845698 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:14.431445 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:14.431315 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:15.151590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:15.144035 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:16.010169 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:16.010130 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:17.000810 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:17.000780 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:17.671630 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:17.671555 4852 generic.go:356] "Generic (PLEG): container finished" podID="1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" containerID="7373d1b8c9784e142253cb4d2cb5ab006a4159586628f7ea493d25eae8604db7" exitCode=0 Nov 04 01:32:17.671826 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:17.671631 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" event={"ID":"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a","Type":"ContainerDied","Data":"7373d1b8c9784e142253cb4d2cb5ab006a4159586628f7ea493d25eae8604db7"} Nov 04 01:32:17.672105 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:17.672087 4852 scope.go:122] "RemoveContainer" containerID="7373d1b8c9784e142253cb4d2cb5ab006a4159586628f7ea493d25eae8604db7" Nov 04 01:32:17.674046 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:17.674016 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:32:17.674116 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:17.674086 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:32:17.674153 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:17.674117 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:32:18.047863 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.047830 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:18.113035 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.113002 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:32:18.232646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.218538 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-envoy\") pod \"207f3c0f-a068-4aee-a593-88e94bcc0dff\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " Nov 04 01:32:18.232646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.218702 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-podinfo\") pod \"207f3c0f-a068-4aee-a593-88e94bcc0dff\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " Nov 04 01:32:18.232646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.218783 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-workload-socket\") pod \"207f3c0f-a068-4aee-a593-88e94bcc0dff\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " Nov 04 01:32:18.232646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.218839 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-credential-socket\") pod \"207f3c0f-a068-4aee-a593-88e94bcc0dff\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " Nov 04 01:32:18.232646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.218960 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/207f3c0f-a068-4aee-a593-88e94bcc0dff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207f3c0f-a068-4aee-a593-88e94bcc0dff-istiod-ca-cert\") pod \"207f3c0f-a068-4aee-a593-88e94bcc0dff\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " Nov 04 01:32:18.232646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.219012 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-workload-certs\") pod \"207f3c0f-a068-4aee-a593-88e94bcc0dff\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " Nov 04 01:32:18.232646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.219131 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/207f3c0f-a068-4aee-a593-88e94bcc0dff-kube-api-access-9s6lg\" (UniqueName: \"kubernetes.io/projected/207f3c0f-a068-4aee-a593-88e94bcc0dff-kube-api-access-9s6lg\") pod \"207f3c0f-a068-4aee-a593-88e94bcc0dff\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " Nov 04 01:32:18.232646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.219169 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-token\" (UniqueName: \"kubernetes.io/projected/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-token\") pod \"207f3c0f-a068-4aee-a593-88e94bcc0dff\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " Nov 04 01:32:18.232646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.219219 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-data\") pod \"207f3c0f-a068-4aee-a593-88e94bcc0dff\" (UID: \"207f3c0f-a068-4aee-a593-88e94bcc0dff\") " Nov 04 01:32:18.284858 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.266964 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-workload-certs" pod "207f3c0f-a068-4aee-a593-88e94bcc0dff" (UID: "207f3c0f-a068-4aee-a593-88e94bcc0dff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:18.284858 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.267163 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-workload-socket" pod "207f3c0f-a068-4aee-a593-88e94bcc0dff" (UID: "207f3c0f-a068-4aee-a593-88e94bcc0dff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:18.315613 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.296135 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-credential-socket" pod "207f3c0f-a068-4aee-a593-88e94bcc0dff" (UID: "207f3c0f-a068-4aee-a593-88e94bcc0dff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:18.315613 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.302647 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-data" pod "207f3c0f-a068-4aee-a593-88e94bcc0dff" (UID: "207f3c0f-a068-4aee-a593-88e94bcc0dff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:18.315613 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.302854 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/207f3c0f-a068-4aee-a593-88e94bcc0dff-istiod-ca-cert" pod "207f3c0f-a068-4aee-a593-88e94bcc0dff" (UID: "207f3c0f-a068-4aee-a593-88e94bcc0dff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:18.442673 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.372431 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207f3c0f-a068-4aee-a593-88e94bcc0dff-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:18.442673 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.372472 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:18.442673 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.372487 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:18.442673 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.372499 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:18.442673 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.372513 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:18.442673 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.414980 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-envoy" pod "207f3c0f-a068-4aee-a593-88e94bcc0dff" (UID: "207f3c0f-a068-4aee-a593-88e94bcc0dff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:18.442673 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.430747 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207f3c0f-a068-4aee-a593-88e94bcc0dff-kube-api-access-9s6lg" pod "207f3c0f-a068-4aee-a593-88e94bcc0dff" (UID: "207f3c0f-a068-4aee-a593-88e94bcc0dff"). InnerVolumeSpecName "kube-api-access-9s6lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:18.467250 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.467190 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-token" pod "207f3c0f-a068-4aee-a593-88e94bcc0dff" (UID: "207f3c0f-a068-4aee-a593-88e94bcc0dff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:18.469134 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.469063 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:18.480794 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.474710 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-podinfo" pod "207f3c0f-a068-4aee-a593-88e94bcc0dff" (UID: "207f3c0f-a068-4aee-a593-88e94bcc0dff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:32:18.604208 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.603405 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s6lg\" (UniqueName: \"kubernetes.io/projected/207f3c0f-a068-4aee-a593-88e94bcc0dff-kube-api-access-9s6lg\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:18.604208 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.603439 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:18.604208 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.603453 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:18.604208 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.603468 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207f3c0f-a068-4aee-a593-88e94bcc0dff-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:18.796297 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.796250 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:18.804612 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.804531 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" event={"ID":"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a","Type":"ContainerStarted","Data":"49d908083b7dcf8a18fe7782318c86de1b2921b47748c0d4edaff508698539f6"} Nov 04 01:32:18.814237 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.814215 4852 generic.go:356] "Generic (PLEG): container finished" podID="207f3c0f-a068-4aee-a593-88e94bcc0dff" containerID="9885f7bd471fdded0f006a03ede0c9f0092a4c5fd9784620552b3ac8e8a677f7" exitCode=0 Nov 04 01:32:18.814520 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.814495 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" event={"ID":"207f3c0f-a068-4aee-a593-88e94bcc0dff","Type":"ContainerDied","Data":"9885f7bd471fdded0f006a03ede0c9f0092a4c5fd9784620552b3ac8e8a677f7"} Nov 04 01:32:18.814642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.814632 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" event={"ID":"207f3c0f-a068-4aee-a593-88e94bcc0dff","Type":"ContainerDied","Data":"f2f3baee6afcaaf47e891fa12882c13cc7b38cff0141be1f518b3278fbd5d87f"} Nov 04 01:32:18.814725 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.814717 4852 scope.go:122] "RemoveContainer" containerID="9885f7bd471fdded0f006a03ede0c9f0092a4c5fd9784620552b3ac8e8a677f7" Nov 04 01:32:18.814913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.814902 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7" Nov 04 01:32:18.830306 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.830275 4852 scope.go:122] "RemoveContainer" containerID="28dc1bfb2ceae7411142f3726755faf4c03092336b4000df1e7d2683547abe2d" Nov 04 01:32:18.848308 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.848278 4852 scope.go:122] "RemoveContainer" containerID="9885f7bd471fdded0f006a03ede0c9f0092a4c5fd9784620552b3ac8e8a677f7" Nov 04 01:32:18.848718 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:32:18.848694 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9885f7bd471fdded0f006a03ede0c9f0092a4c5fd9784620552b3ac8e8a677f7\": container with ID starting with 9885f7bd471fdded0f006a03ede0c9f0092a4c5fd9784620552b3ac8e8a677f7 not found: ID does not exist" containerID="9885f7bd471fdded0f006a03ede0c9f0092a4c5fd9784620552b3ac8e8a677f7" Nov 04 01:32:18.848845 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.848721 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9885f7bd471fdded0f006a03ede0c9f0092a4c5fd9784620552b3ac8e8a677f7"} err="failed to get container status \"9885f7bd471fdded0f006a03ede0c9f0092a4c5fd9784620552b3ac8e8a677f7\": rpc error: code = NotFound desc = could not find container \"9885f7bd471fdded0f006a03ede0c9f0092a4c5fd9784620552b3ac8e8a677f7\": container with ID starting with 9885f7bd471fdded0f006a03ede0c9f0092a4c5fd9784620552b3ac8e8a677f7 not found: ID does not exist" Nov 04 01:32:18.848845 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.848743 4852 scope.go:122] "RemoveContainer" containerID="28dc1bfb2ceae7411142f3726755faf4c03092336b4000df1e7d2683547abe2d" Nov 04 01:32:18.849205 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:32:18.849177 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28dc1bfb2ceae7411142f3726755faf4c03092336b4000df1e7d2683547abe2d\": container with ID starting with 28dc1bfb2ceae7411142f3726755faf4c03092336b4000df1e7d2683547abe2d not found: ID does not exist" containerID="28dc1bfb2ceae7411142f3726755faf4c03092336b4000df1e7d2683547abe2d" Nov 04 01:32:18.849301 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.849209 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28dc1bfb2ceae7411142f3726755faf4c03092336b4000df1e7d2683547abe2d"} err="failed to get container status \"28dc1bfb2ceae7411142f3726755faf4c03092336b4000df1e7d2683547abe2d\": rpc error: code = NotFound desc = could not find container \"28dc1bfb2ceae7411142f3726755faf4c03092336b4000df1e7d2683547abe2d\": container with ID starting with 28dc1bfb2ceae7411142f3726755faf4c03092336b4000df1e7d2683547abe2d not found: ID does not exist" Nov 04 01:32:18.890884 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.890805 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7"] Nov 04 01:32:18.894931 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:18.894904 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-qprv7"] Nov 04 01:32:19.143104 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.143024 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:32:19.143934 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.143904 4852 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:32:19.144079 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.143960 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" podUID="1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 04 01:32:19.447001 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.446921 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc"] Nov 04 01:32:19.447649 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.447634 4852 cpu_manager.go:414] "containerMap: removing container" podUID="158b1659-369b-454e-bb71-db3763ca6774" containerName="istio-proxy" Nov 04 01:32:19.447737 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.447651 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158b1659-369b-454e-bb71-db3763ca6774" containerName="istio-proxy" Nov 04 01:32:19.447737 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.447666 4852 cpu_manager.go:414] "containerMap: removing container" podUID="207f3c0f-a068-4aee-a593-88e94bcc0dff" containerName="istio-proxy" Nov 04 01:32:19.447737 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.447670 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="207f3c0f-a068-4aee-a593-88e94bcc0dff" containerName="istio-proxy" Nov 04 01:32:19.447737 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.447701 4852 cpu_manager.go:414] "containerMap: removing container" podUID="b1aea2bc-8d86-4c48-8ce7-e0182f02328b" containerName="main" Nov 04 01:32:19.447737 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.447706 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1aea2bc-8d86-4c48-8ce7-e0182f02328b" containerName="main" Nov 04 01:32:19.447737 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.447715 4852 cpu_manager.go:414] "containerMap: removing container" podUID="207f3c0f-a068-4aee-a593-88e94bcc0dff" containerName="istio-proxy" Nov 04 01:32:19.447737 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.447720 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="207f3c0f-a068-4aee-a593-88e94bcc0dff" containerName="istio-proxy" Nov 04 01:32:19.447737 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.447728 4852 cpu_manager.go:414] "containerMap: removing container" podUID="85f24a4b-278a-432f-a86b-f3addc378bee" containerName="istio-proxy" Nov 04 01:32:19.447737 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.447733 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f24a4b-278a-432f-a86b-f3addc378bee" containerName="istio-proxy" Nov 04 01:32:19.448021 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.447756 4852 cpu_manager.go:414] "containerMap: removing container" podUID="107a1fd6-feb8-4615-91ef-980f9af78d5f" containerName="istio-proxy" Nov 04 01:32:19.448021 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.447760 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107a1fd6-feb8-4615-91ef-980f9af78d5f" containerName="istio-proxy" Nov 04 01:32:19.448021 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.447847 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="107a1fd6-feb8-4615-91ef-980f9af78d5f" containerName="istio-proxy" Nov 04 01:32:19.448021 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.447856 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="158b1659-369b-454e-bb71-db3763ca6774" containerName="istio-proxy" Nov 04 01:32:19.448021 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.447862 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="207f3c0f-a068-4aee-a593-88e94bcc0dff" containerName="istio-proxy" Nov 04 01:32:19.448021 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.447874 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1aea2bc-8d86-4c48-8ce7-e0182f02328b" containerName="main" Nov 04 01:32:19.448021 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.447879 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f24a4b-278a-432f-a86b-f3addc378bee" containerName="istio-proxy" Nov 04 01:32:19.451267 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.451249 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc" Nov 04 01:32:19.475120 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.463340 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bfc54d\"" Nov 04 01:32:19.488643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.478200 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:19.498802 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.498768 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc"] Nov 04 01:32:19.504868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.504849 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bfc54d-dockercfg-wwpkt\"" Nov 04 01:32:19.510794 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.510767 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts"] Nov 04 01:32:19.512098 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.512074 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="207f3c0f-a068-4aee-a593-88e94bcc0dff" containerName="istio-proxy" Nov 04 01:32:19.515154 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.515058 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts" Nov 04 01:32:19.518283 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.518259 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts"] Nov 04 01:32:19.528508 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.526548 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wt2m\" (UniqueName: \"kubernetes.io/projected/68127737-8431-4191-ad9a-d16e080a2ec5-kube-api-access-5wt2m\") pod \"payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc\" (UID: \"68127737-8431-4191-ad9a-d16e080a2ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc" Nov 04 01:32:19.528508 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.528386 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68127737-8431-4191-ad9a-d16e080a2ec5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc\" (UID: \"68127737-8431-4191-ad9a-d16e080a2ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc" Nov 04 01:32:19.638597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.632479 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68127737-8431-4191-ad9a-d16e080a2ec5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc\" (UID: \"68127737-8431-4191-ad9a-d16e080a2ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc" Nov 04 01:32:19.683087 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.681135 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68127737-8431-4191-ad9a-d16e080a2ec5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc\" (UID: \"68127737-8431-4191-ad9a-d16e080a2ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc" Nov 04 01:32:19.772643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.715434 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:19.772643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.726540 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wt2m\" (UniqueName: \"kubernetes.io/projected/68127737-8431-4191-ad9a-d16e080a2ec5-kube-api-access-5wt2m\") pod \"payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc\" (UID: \"68127737-8431-4191-ad9a-d16e080a2ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc" Nov 04 01:32:19.772643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.741237 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wt2m\" (UniqueName: \"kubernetes.io/projected/68127737-8431-4191-ad9a-d16e080a2ec5-kube-api-access-5wt2m\") pod \"payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc\" (UID: \"68127737-8431-4191-ad9a-d16e080a2ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc" Nov 04 01:32:19.885593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.864270 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl4j6\" (UniqueName: \"kubernetes.io/projected/2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4-kube-api-access-cl4j6\") pod \"payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts\" (UID: \"2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts" Nov 04 01:32:19.885593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.864328 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts\" (UID: \"2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts" Nov 04 01:32:19.899590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.886624 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc" Nov 04 01:32:19.981332 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.976061 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl4j6\" (UniqueName: \"kubernetes.io/projected/2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4-kube-api-access-cl4j6\") pod \"payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts\" (UID: \"2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts" Nov 04 01:32:19.981332 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.976109 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts\" (UID: \"2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts" Nov 04 01:32:20.005210 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:19.998543 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts\" (UID: \"2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts" Nov 04 01:32:20.119136 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:20.119052 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="207f3c0f-a068-4aee-a593-88e94bcc0dff" path="/var/lib/kubelet/pods/207f3c0f-a068-4aee-a593-88e94bcc0dff/volumes" Nov 04 01:32:20.124757 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:20.124716 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:32:20.143513 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:20.143475 4852 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:32:20.156939 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:20.156899 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" podUID="1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 04 01:32:20.157449 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:20.149286 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl4j6\" (UniqueName: \"kubernetes.io/projected/2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4-kube-api-access-cl4j6\") pod \"payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts\" (UID: \"2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts" Nov 04 01:32:20.184617 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:20.179545 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:20.434836 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:20.434760 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts" Nov 04 01:32:20.591720 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:20.591676 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc"] Nov 04 01:32:20.597770 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:32:20.597742 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68127737_8431_4191_ad9a_d16e080a2ec5.slice/crio-ca004a5c30c7092386e16d63f09b8a9f2a94637cf11cb3420c79cb42f48be29b WatchSource:0}: Error finding container ca004a5c30c7092386e16d63f09b8a9f2a94637cf11cb3420c79cb42f48be29b: Status 404 returned error can't find the container with id ca004a5c30c7092386e16d63f09b8a9f2a94637cf11cb3420c79cb42f48be29b Nov 04 01:32:20.753137 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:32:20.753103 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d752c6b_ef0d_4eeb_a1de_10f3e889f0f4.slice/crio-de54f8544006673bf05119b177312589c3cb973717981ac5561e498e31fe2962 WatchSource:0}: Error finding container de54f8544006673bf05119b177312589c3cb973717981ac5561e498e31fe2962: Status 404 returned error can't find the container with id de54f8544006673bf05119b177312589c3cb973717981ac5561e498e31fe2962 Nov 04 01:32:20.754185 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:20.754164 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts"] Nov 04 01:32:20.987641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:20.986386 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts" event={"ID":"2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4","Type":"ContainerStarted","Data":"8dc8cf3b3f7d5550644d81a87c0e04f205f3ee6fcce5c2a6ff116955c893ae90"} Nov 04 01:32:20.987641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:20.986489 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts" event={"ID":"2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4","Type":"ContainerStarted","Data":"de54f8544006673bf05119b177312589c3cb973717981ac5561e498e31fe2962"} Nov 04 01:32:20.988945 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:20.988903 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts" Nov 04 01:32:20.990294 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:20.990268 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc" event={"ID":"68127737-8431-4191-ad9a-d16e080a2ec5","Type":"ContainerStarted","Data":"fab2c9d069534d4e7bc70e93ae3b1ef293c1bf1f11eb958a6f40eb141fd19dc5"} Nov 04 01:32:20.990402 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:20.990332 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc" event={"ID":"68127737-8431-4191-ad9a-d16e080a2ec5","Type":"ContainerStarted","Data":"ca004a5c30c7092386e16d63f09b8a9f2a94637cf11cb3420c79cb42f48be29b"} Nov 04 01:32:20.990402 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:20.990366 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc" Nov 04 01:32:21.014313 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:21.014270 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts" podStartSLOduration=2.014257367 podStartE2EDuration="2.014257367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:21.009097758 +0000 UTC m=+2023.124207870" watchObservedRunningTime="2026-11-04 01:32:21.014257367 +0000 UTC m=+2023.129367485" Nov 04 01:32:21.030472 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:21.030430 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc" podStartSLOduration=2.030412859 podStartE2EDuration="2.030412859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:21.029158143 +0000 UTC m=+2023.144268261" watchObservedRunningTime="2026-11-04 01:32:21.030412859 +0000 UTC m=+2023.145522973" Nov 04 01:32:21.144516 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:21.144486 4852 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:32:21.144950 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:21.144539 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" podUID="1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 04 01:32:21.238218 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:21.238100 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:21.407832 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:21.407799 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:22.086677 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.086641 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl"] Nov 04 01:32:22.090231 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.090211 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.098046 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.098023 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-50dab0-openshift-default-dockercfg-nmfg2\"" Nov 04 01:32:22.107773 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.107747 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl"] Nov 04 01:32:22.143413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.143384 4852 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:32:22.143623 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.143427 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" podUID="1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 04 01:32:22.197647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.195839 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-data\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.197647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.196047 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istiod-ca-cert\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.197647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.196286 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-token\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.197647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.196510 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-credential-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.197647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.196667 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-podinfo\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.197647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.196694 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjgd5\" (UniqueName: \"kubernetes.io/projected/dbba8e43-9895-4f2e-b1a1-a91208543bf1-kube-api-access-kjgd5\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.197647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.196759 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-envoy\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.197647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.196778 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-workload-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.197647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.196839 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-workload-certs\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.298864 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.298826 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-credential-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.299029 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.298923 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-podinfo\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.299029 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.298948 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjgd5\" (UniqueName: \"kubernetes.io/projected/dbba8e43-9895-4f2e-b1a1-a91208543bf1-kube-api-access-kjgd5\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.299029 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.298996 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-envoy\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.299178 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.299021 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-workload-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.299178 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.299057 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-workload-certs\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.299300 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.299187 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-data\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.299300 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.299249 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-credential-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.299300 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.299259 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istiod-ca-cert\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.299504 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.299425 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-workload-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.299504 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.299458 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-token\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.299504 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.299484 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-workload-certs\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.299698 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.299539 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-data\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.299955 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.299910 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istiod-ca-cert\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.301508 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.301485 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-envoy\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.301786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.301765 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-podinfo\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.308138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.308112 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-token\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.308509 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.308486 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjgd5\" (UniqueName: \"kubernetes.io/projected/dbba8e43-9895-4f2e-b1a1-a91208543bf1-kube-api-access-kjgd5\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.404723 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.404636 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:22.491237 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:22.472106 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:23.016595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:23.015103 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:23.033872 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:23.021630 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl"] Nov 04 01:32:23.104325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:23.104280 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:32:23.104453 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:23.104406 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:32:23.104514 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:23.104479 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:32:23.144404 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:23.144378 4852 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:32:23.144495 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:23.144432 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" podUID="1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 04 01:32:23.642592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:23.642446 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:23.859626 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:23.856824 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:23.927520 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:23.927437 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk"] Nov 04 01:32:23.931989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:23.931965 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:23.946162 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:23.944112 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8e9791-openshift-default-dockercfg-zzlnh\"" Nov 04 01:32:23.947014 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:23.946988 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk"] Nov 04 01:32:24.004850 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.004812 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-workload-socket\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.005045 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.004883 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-credential-socket\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.005045 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.004914 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-podinfo\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.005045 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.004958 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t46jl\" (UniqueName: \"kubernetes.io/projected/6604612f-7353-40ab-8a1b-29b19b3395f0-kube-api-access-t46jl\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.005045 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.005009 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-token\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.005045 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.005029 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6604612f-7353-40ab-8a1b-29b19b3395f0-istiod-ca-cert\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.005299 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.005068 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-envoy\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.005299 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.005102 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-data\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.005299 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.005155 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-workload-certs\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.145590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.119816 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-workload-socket\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.145590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.119951 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-credential-socket\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.145590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.120019 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-podinfo\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.145590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.120058 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t46jl\" (UniqueName: \"kubernetes.io/projected/6604612f-7353-40ab-8a1b-29b19b3395f0-kube-api-access-t46jl\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.145590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.120169 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-token\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.145590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.120205 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6604612f-7353-40ab-8a1b-29b19b3395f0-istiod-ca-cert\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.145590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.120289 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-envoy\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.145590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.120345 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-data\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.145590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.120495 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-workload-certs\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.145590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.144217 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-workload-socket\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.187650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.147532 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6604612f-7353-40ab-8a1b-29b19b3395f0-istiod-ca-cert\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.187650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.155351 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-envoy\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.187650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.155641 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-data\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.187650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.156136 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-workload-certs\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.191757 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.190551 4852 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:32:24.191757 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.190626 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" podUID="1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 04 01:32:24.210876 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.210739 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-podinfo\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.215601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.212086 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:24.253269 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.253239 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-token\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.274303 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.262201 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" event={"ID":"dbba8e43-9895-4f2e-b1a1-a91208543bf1","Type":"ContainerStarted","Data":"680e02a4ad15ee2e3f3a7cefc134176ab63ef5d6e4434bc7dbf3c8d801d7a194"} Nov 04 01:32:24.274581 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.274537 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" event={"ID":"dbba8e43-9895-4f2e-b1a1-a91208543bf1","Type":"ContainerStarted","Data":"3aba1d1e6bb568590a31eede0c830a5b0353106c985832072340f3f8c9bc0581"} Nov 04 01:32:24.332546 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.332476 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" podStartSLOduration=2.332451773 podStartE2EDuration="2.332451773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:24.317010863 +0000 UTC m=+2026.432120981" watchObservedRunningTime="2026-11-04 01:32:24.332451773 +0000 UTC m=+2026.447561892" Nov 04 01:32:24.483644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.457718 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-credential-socket\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.483644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.458548 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t46jl\" (UniqueName: \"kubernetes.io/projected/6604612f-7353-40ab-8a1b-29b19b3395f0-kube-api-access-t46jl\") pod \"e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.488500 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.488462 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:24.506970 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.506945 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:24.583587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.579494 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:24.588596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.583185 4852 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 04 01:32:24.588758 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:24.588599 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" podUID="dbba8e43-9895-4f2e-b1a1-a91208543bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 04 01:32:25.049234 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:25.049144 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:25.153013 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:25.152969 4852 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:32:25.153341 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:25.153270 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" podUID="1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 04 01:32:25.216234 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:25.216193 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:25.273591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:25.247758 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:25.409962 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:25.409863 4852 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 04 01:32:25.410250 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:25.410198 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" podUID="dbba8e43-9895-4f2e-b1a1-a91208543bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 04 01:32:25.606067 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:25.605673 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:25.608010 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:25.607944 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:25.623890 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:25.623854 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:25.858045 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:25.858005 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk"] Nov 04 01:32:25.893477 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:25.893437 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:32:25.895330 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:25.895299 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:32:25.895757 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:25.895733 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:32:26.143766 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:26.143670 4852 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:32:26.144282 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:26.144254 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" podUID="1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 04 01:32:26.239247 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:26.233931 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:26.425640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:26.420927 4852 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 04 01:32:26.425640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:26.420990 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" podUID="dbba8e43-9895-4f2e-b1a1-a91208543bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 04 01:32:26.447667 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:26.447635 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" event={"ID":"6604612f-7353-40ab-8a1b-29b19b3395f0","Type":"ContainerStarted","Data":"61ba8bb1d7fb264cb90f65de8d21537ccc1d63ee031575b78765002f4e599288"} Nov 04 01:32:26.447923 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:26.447900 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" event={"ID":"6604612f-7353-40ab-8a1b-29b19b3395f0","Type":"ContainerStarted","Data":"16835452ef1317104f7ed86be665878e979ce729415922e9e26ad28124d8b1da"} Nov 04 01:32:26.506306 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:26.506234 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" podStartSLOduration=3.506213795 podStartE2EDuration="3.506213795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:26.49304864 +0000 UTC m=+2028.608158759" watchObservedRunningTime="2026-11-04 01:32:26.506213795 +0000 UTC m=+2028.621323914" Nov 04 01:32:26.586588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:26.580636 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:26.802593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:26.801435 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:27.034593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:26.993920 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:27.181104 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.180972 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:27.413897 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.413859 4852 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 04 01:32:27.414142 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.414114 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" podUID="dbba8e43-9895-4f2e-b1a1-a91208543bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 04 01:32:27.590590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.589921 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:32:27.594947 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.594768 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:32:27.601838 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.601814 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:27.610212 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.606833 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h"] Nov 04 01:32:27.614599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.614550 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h"] Nov 04 01:32:27.614730 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.614595 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" Nov 04 01:32:27.630012 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.629982 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c75ea1-serving-cert\"" Nov 04 01:32:27.736128 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.736089 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf3ef542-79a2-485f-b8ae-0abd09b05734-maas-api-tls\") pod \"maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h\" (UID: \"cf3ef542-79a2-485f-b8ae-0abd09b05734\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" Nov 04 01:32:27.736320 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.736157 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf3ef542-79a2-485f-b8ae-0abd09b05734-openshift-service-ca\") pod \"maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h\" (UID: \"cf3ef542-79a2-485f-b8ae-0abd09b05734\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" Nov 04 01:32:27.736320 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.736219 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf3ef542-79a2-485f-b8ae-0abd09b05734-metrics-certs\") pod \"maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h\" (UID: \"cf3ef542-79a2-485f-b8ae-0abd09b05734\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" Nov 04 01:32:27.736458 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.736364 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8xtg\" (UniqueName: \"kubernetes.io/projected/cf3ef542-79a2-485f-b8ae-0abd09b05734-kube-api-access-t8xtg\") pod \"maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h\" (UID: \"cf3ef542-79a2-485f-b8ae-0abd09b05734\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" Nov 04 01:32:27.748540 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.746960 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:27.844683 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.838806 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf3ef542-79a2-485f-b8ae-0abd09b05734-maas-api-tls\") pod \"maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h\" (UID: \"cf3ef542-79a2-485f-b8ae-0abd09b05734\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" Nov 04 01:32:27.844683 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.838877 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf3ef542-79a2-485f-b8ae-0abd09b05734-openshift-service-ca\") pod \"maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h\" (UID: \"cf3ef542-79a2-485f-b8ae-0abd09b05734\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" Nov 04 01:32:27.844683 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.838943 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf3ef542-79a2-485f-b8ae-0abd09b05734-metrics-certs\") pod \"maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h\" (UID: \"cf3ef542-79a2-485f-b8ae-0abd09b05734\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" Nov 04 01:32:27.844683 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.839077 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8xtg\" (UniqueName: \"kubernetes.io/projected/cf3ef542-79a2-485f-b8ae-0abd09b05734-kube-api-access-t8xtg\") pod \"maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h\" (UID: \"cf3ef542-79a2-485f-b8ae-0abd09b05734\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" Nov 04 01:32:27.844683 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:32:27.839713 4852 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-serving-cert: secret "maas-api-e2e-api-b-c75ea1-serving-cert" not found Nov 04 01:32:27.844683 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:32:27.839807 4852 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf3ef542-79a2-485f-b8ae-0abd09b05734-maas-api-tls podName:cf3ef542-79a2-485f-b8ae-0abd09b05734 nodeName:}" failed. No retries permitted until 2026-11-04 01:32:28.339784366 +0000 UTC m=+2030.454894464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf3ef542-79a2-485f-b8ae-0abd09b05734-maas-api-tls") pod "maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" (UID: "cf3ef542-79a2-485f-b8ae-0abd09b05734") : secret "maas-api-e2e-api-b-c75ea1-serving-cert" not found Nov 04 01:32:27.844683 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.840767 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf3ef542-79a2-485f-b8ae-0abd09b05734-openshift-service-ca\") pod \"maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h\" (UID: \"cf3ef542-79a2-485f-b8ae-0abd09b05734\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" Nov 04 01:32:27.909600 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.903815 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8xtg\" (UniqueName: \"kubernetes.io/projected/cf3ef542-79a2-485f-b8ae-0abd09b05734-kube-api-access-t8xtg\") pod \"maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h\" (UID: \"cf3ef542-79a2-485f-b8ae-0abd09b05734\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" Nov 04 01:32:27.915256 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.915186 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf3ef542-79a2-485f-b8ae-0abd09b05734-metrics-certs\") pod \"maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h\" (UID: \"cf3ef542-79a2-485f-b8ae-0abd09b05734\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" Nov 04 01:32:27.922123 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:27.922039 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:28.371583 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:28.369225 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf3ef542-79a2-485f-b8ae-0abd09b05734-maas-api-tls\") pod \"maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h\" (UID: \"cf3ef542-79a2-485f-b8ae-0abd09b05734\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" Nov 04 01:32:28.407858 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:28.403821 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf3ef542-79a2-485f-b8ae-0abd09b05734-maas-api-tls\") pod \"maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h\" (UID: \"cf3ef542-79a2-485f-b8ae-0abd09b05734\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" Nov 04 01:32:28.407858 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:28.406494 4852 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 04 01:32:28.407858 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:28.406604 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" podUID="dbba8e43-9895-4f2e-b1a1-a91208543bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 04 01:32:28.461601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:28.458127 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:28.520673 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:28.513683 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:28.528822 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:28.528794 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:32:28.626638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:28.624667 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" Nov 04 01:32:29.037507 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:29.037478 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:29.328640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:29.328092 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:29.441600 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:29.438741 4852 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 04 01:32:29.441600 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:29.438801 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" podUID="dbba8e43-9895-4f2e-b1a1-a91208543bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 04 01:32:29.614864 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:29.614834 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h"] Nov 04 01:32:29.633337 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:29.633291 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:29.895534 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:32:29.895438 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:32:30.405811 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:30.405770 4852 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 04 01:32:30.405986 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:30.405831 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" podUID="dbba8e43-9895-4f2e-b1a1-a91208543bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 04 01:32:30.749769 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:30.747961 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:30.840915 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:30.840860 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" event={"ID":"cf3ef542-79a2-485f-b8ae-0abd09b05734","Type":"ContainerStarted","Data":"6f1a9aa277768bbd17d4c744e6f90ee9b84a8ca48b21bdd3cd65efc411cea5e0"} Nov 04 01:32:31.152883 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:31.152831 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:31.400588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:31.391755 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:31.411648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:31.406920 4852 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 04 01:32:31.411648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:31.406980 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" podUID="dbba8e43-9895-4f2e-b1a1-a91208543bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 04 01:32:31.853699 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:31.853499 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" event={"ID":"cf3ef542-79a2-485f-b8ae-0abd09b05734","Type":"ContainerStarted","Data":"3a42f37bf986f629b2b88b5b90555913d4ab8aba590b0c9d0ecc240d24a7bfc8"} Nov 04 01:32:31.853699 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:31.853648 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" Nov 04 01:32:31.901614 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:31.900008 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:32.040240 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:32.031163 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts" Nov 04 01:32:32.040240 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:32.031244 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc" Nov 04 01:32:32.085588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:32.080361 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" podStartSLOduration=3.936672753 podStartE2EDuration="5.080339473s" totalImagesPullingTime="1.14366672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:31.887821794 +0000 UTC m=+2034.002931939" watchObservedRunningTime="2026-11-04 01:32:32.080339473 +0000 UTC m=+2034.195449592" Nov 04 01:32:32.405173 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:32.405139 4852 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 04 01:32:32.405362 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:32.405196 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" podUID="dbba8e43-9895-4f2e-b1a1-a91208543bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 04 01:32:32.710796 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:32.710072 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:33.410714 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:33.410687 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:33.411134 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:33.411094 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:33.411742 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:33.411722 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:33.490967 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:33.490934 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:33.676546 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:33.676456 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:34.476365 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:34.476336 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:35.368474 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:35.368440 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:35.697929 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:35.697840 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:35.883594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:35.870536 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:36.539158 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:36.539112 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:37.105429 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.105384 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc"] Nov 04 01:32:37.110230 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.110206 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.125675 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.125652 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc"] Nov 04 01:32:37.248624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.248589 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bblfx\" (UniqueName: \"kubernetes.io/projected/f244db1e-5289-4885-9e1c-2b63731821ca-kube-api-access-bblfx\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.248784 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.248640 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-workload-certs\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.248784 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.248725 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f244db1e-5289-4885-9e1c-2b63731821ca-istio-token\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.248926 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.248788 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-credential-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.248926 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.248841 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f244db1e-5289-4885-9e1c-2b63731821ca-istio-podinfo\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.248926 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.248917 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-istio-envoy\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.249066 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.248935 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f244db1e-5289-4885-9e1c-2b63731821ca-istiod-ca-cert\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.249145 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.249118 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-workload-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.249226 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.249210 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-istio-data\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.358643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.355381 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-istio-envoy\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.358643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.355439 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f244db1e-5289-4885-9e1c-2b63731821ca-istiod-ca-cert\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.358643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.355516 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-workload-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.358643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.355593 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-istio-data\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.358643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.355800 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bblfx\" (UniqueName: \"kubernetes.io/projected/f244db1e-5289-4885-9e1c-2b63731821ca-kube-api-access-bblfx\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.358643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.355831 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-workload-certs\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.358643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.355866 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f244db1e-5289-4885-9e1c-2b63731821ca-istio-token\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.358643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.355933 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-credential-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.358643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.355965 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f244db1e-5289-4885-9e1c-2b63731821ca-istio-podinfo\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.358643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.357298 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-istio-data\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.382596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.373735 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-workload-certs\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.382596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.374226 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-credential-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.382596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.374687 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f244db1e-5289-4885-9e1c-2b63731821ca-istio-podinfo\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.382596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.377982 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-workload-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.382596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.378654 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f244db1e-5289-4885-9e1c-2b63731821ca-istiod-ca-cert\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.382596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.380545 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:37.421490 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.419937 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-istio-envoy\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.421490 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.420403 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f244db1e-5289-4885-9e1c-2b63731821ca-istio-token\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.421490 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.421122 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bblfx\" (UniqueName: \"kubernetes.io/projected/f244db1e-5289-4885-9e1c-2b63731821ca-kube-api-access-bblfx\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.442654 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.442624 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:37.737409 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.737380 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc"] Nov 04 01:32:37.737744 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:32:37.737718 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf244db1e_5289_4885_9e1c_2b63731821ca.slice/crio-0e2aef6de0602b5beed56efefc7eaf0d85e6610a352325da93918c42825c5754 WatchSource:0}: Error finding container 0e2aef6de0602b5beed56efefc7eaf0d85e6610a352325da93918c42825c5754: Status 404 returned error can't find the container with id 0e2aef6de0602b5beed56efefc7eaf0d85e6610a352325da93918c42825c5754 Nov 04 01:32:37.740656 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.740624 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:32:37.740725 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.740686 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:32:37.740725 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.740717 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:32:37.872590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.870986 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" Nov 04 01:32:37.891506 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.889121 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:37.891506 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.891307 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx"] Nov 04 01:32:37.977199 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.977162 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" event={"ID":"f244db1e-5289-4885-9e1c-2b63731821ca","Type":"ContainerStarted","Data":"b165ff6f0416d8ecf3a958cf04cd7c4a8ccff0226ef2d3b6024fadb83f510451"} Nov 04 01:32:37.977406 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.977227 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" event={"ID":"f244db1e-5289-4885-9e1c-2b63731821ca","Type":"ContainerStarted","Data":"0e2aef6de0602b5beed56efefc7eaf0d85e6610a352325da93918c42825c5754"} Nov 04 01:32:37.977523 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:37.977501 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" podUID="4b90a88e-68fc-4841-b289-5af7080408d6" containerName="maas-api" containerID="cri-o://b2fe84c03406329972b52b0a5d64da509f76e6126f1875cf62e94004b4efad49" gracePeriod=30 Nov 04 01:32:38.006429 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:38.005308 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" podStartSLOduration=1.005288477 podStartE2EDuration="1.005288477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:38.001212463 +0000 UTC m=+2040.116322621" watchObservedRunningTime="2026-11-04 01:32:38.005288477 +0000 UTC m=+2040.120398597" Nov 04 01:32:38.043839 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:38.043801 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:38.318598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:38.316089 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:38.456594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:38.450311 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:38.456594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:38.452498 4852 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 04 01:32:38.456594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:38.452549 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" podUID="f244db1e-5289-4885-9e1c-2b63731821ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 04 01:32:38.909244 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:38.909219 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" Nov 04 01:32:38.997193 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:38.997099 4852 generic.go:356] "Generic (PLEG): container finished" podID="4b90a88e-68fc-4841-b289-5af7080408d6" containerID="b2fe84c03406329972b52b0a5d64da509f76e6126f1875cf62e94004b4efad49" exitCode=0 Nov 04 01:32:38.997368 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:38.997270 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" Nov 04 01:32:38.997368 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:38.997338 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" event={"ID":"4b90a88e-68fc-4841-b289-5af7080408d6","Type":"ContainerDied","Data":"b2fe84c03406329972b52b0a5d64da509f76e6126f1875cf62e94004b4efad49"} Nov 04 01:32:38.997536 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:38.997379 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx" event={"ID":"4b90a88e-68fc-4841-b289-5af7080408d6","Type":"ContainerDied","Data":"53c18586de56a2eb718b913c22ec6e16d14edc5ff4ab2cfedd1facbe4e00e3a1"} Nov 04 01:32:38.997536 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:38.997409 4852 scope.go:122] "RemoveContainer" containerID="b2fe84c03406329972b52b0a5d64da509f76e6126f1875cf62e94004b4efad49" Nov 04 01:32:39.015986 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:39.015947 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b90a88e-68fc-4841-b289-5af7080408d6-kube-api-access-jnb9z\" (UniqueName: \"kubernetes.io/projected/4b90a88e-68fc-4841-b289-5af7080408d6-kube-api-access-jnb9z\") pod \"4b90a88e-68fc-4841-b289-5af7080408d6\" (UID: \"4b90a88e-68fc-4841-b289-5af7080408d6\") " Nov 04 01:32:39.016163 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:39.016114 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b90a88e-68fc-4841-b289-5af7080408d6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b90a88e-68fc-4841-b289-5af7080408d6-maas-api-tls\") pod \"4b90a88e-68fc-4841-b289-5af7080408d6\" (UID: \"4b90a88e-68fc-4841-b289-5af7080408d6\") " Nov 04 01:32:39.016291 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:39.016198 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b90a88e-68fc-4841-b289-5af7080408d6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b90a88e-68fc-4841-b289-5af7080408d6-openshift-service-ca\") pod \"4b90a88e-68fc-4841-b289-5af7080408d6\" (UID: \"4b90a88e-68fc-4841-b289-5af7080408d6\") " Nov 04 01:32:39.016400 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:39.016381 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b90a88e-68fc-4841-b289-5af7080408d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b90a88e-68fc-4841-b289-5af7080408d6-metrics-certs\") pod \"4b90a88e-68fc-4841-b289-5af7080408d6\" (UID: \"4b90a88e-68fc-4841-b289-5af7080408d6\") " Nov 04 01:32:39.020469 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:39.020259 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b90a88e-68fc-4841-b289-5af7080408d6-openshift-service-ca" pod "4b90a88e-68fc-4841-b289-5af7080408d6" (UID: "4b90a88e-68fc-4841-b289-5af7080408d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:39.022807 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:39.022421 4852 scope.go:122] "RemoveContainer" containerID="b2fe84c03406329972b52b0a5d64da509f76e6126f1875cf62e94004b4efad49" Nov 04 01:32:39.022927 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:32:39.022865 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2fe84c03406329972b52b0a5d64da509f76e6126f1875cf62e94004b4efad49\": container with ID starting with b2fe84c03406329972b52b0a5d64da509f76e6126f1875cf62e94004b4efad49 not found: ID does not exist" containerID="b2fe84c03406329972b52b0a5d64da509f76e6126f1875cf62e94004b4efad49" Nov 04 01:32:39.022927 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:39.022904 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2fe84c03406329972b52b0a5d64da509f76e6126f1875cf62e94004b4efad49"} err="failed to get container status \"b2fe84c03406329972b52b0a5d64da509f76e6126f1875cf62e94004b4efad49\": rpc error: code = NotFound desc = could not find container \"b2fe84c03406329972b52b0a5d64da509f76e6126f1875cf62e94004b4efad49\": container with ID starting with b2fe84c03406329972b52b0a5d64da509f76e6126f1875cf62e94004b4efad49 not found: ID does not exist" Nov 04 01:32:39.024127 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:39.024095 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b90a88e-68fc-4841-b289-5af7080408d6-maas-api-tls" pod "4b90a88e-68fc-4841-b289-5af7080408d6" (UID: "4b90a88e-68fc-4841-b289-5af7080408d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:39.027059 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:39.026997 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b90a88e-68fc-4841-b289-5af7080408d6-metrics-certs" pod "4b90a88e-68fc-4841-b289-5af7080408d6" (UID: "4b90a88e-68fc-4841-b289-5af7080408d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:39.048823 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:39.048784 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b90a88e-68fc-4841-b289-5af7080408d6-kube-api-access-jnb9z" pod "4b90a88e-68fc-4841-b289-5af7080408d6" (UID: "4b90a88e-68fc-4841-b289-5af7080408d6"). InnerVolumeSpecName "kube-api-access-jnb9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:39.089689 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:39.089547 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:39.119578 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:39.119544 4852 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b90a88e-68fc-4841-b289-5af7080408d6-metrics-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:39.119706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:39.119582 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnb9z\" (UniqueName: \"kubernetes.io/projected/4b90a88e-68fc-4841-b289-5af7080408d6-kube-api-access-jnb9z\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:39.119706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:39.119593 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b90a88e-68fc-4841-b289-5af7080408d6-maas-api-tls\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:39.119706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:39.119602 4852 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b90a88e-68fc-4841-b289-5af7080408d6-openshift-service-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:39.245160 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:39.244200 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:39.341702 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:39.341630 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx"] Nov 04 01:32:39.357703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:39.350991 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-50dab0-86579ff588-pmkvx"] Nov 04 01:32:39.443873 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:39.443837 4852 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 04 01:32:39.444055 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:39.443891 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" podUID="f244db1e-5289-4885-9e1c-2b63731821ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 04 01:32:39.925591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:39.925005 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:40.036587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:40.032922 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b90a88e-68fc-4841-b289-5af7080408d6" path="/var/lib/kubelet/pods/4b90a88e-68fc-4841-b289-5af7080408d6/volumes" Nov 04 01:32:40.105586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:40.105433 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:40.321174 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:40.316446 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:40.447726 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:40.445973 4852 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 04 01:32:40.447726 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:40.446047 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" podUID="f244db1e-5289-4885-9e1c-2b63731821ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 04 01:32:40.991879 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:40.991670 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:41.303151 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:41.303121 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl"] Nov 04 01:32:41.303727 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:41.303692 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" podUID="dbba8e43-9895-4f2e-b1a1-a91208543bf1" containerName="istio-proxy" containerID="cri-o://680e02a4ad15ee2e3f3a7cefc134176ab63ef5d6e4434bc7dbf3c8d801d7a194" gracePeriod=30 Nov 04 01:32:41.319944 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:41.319917 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc"] Nov 04 01:32:41.320178 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:41.320148 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" podUID="f244db1e-5289-4885-9e1c-2b63731821ca" containerName="istio-proxy" containerID="cri-o://b165ff6f0416d8ecf3a958cf04cd7c4a8ccff0226ef2d3b6024fadb83f510451" gracePeriod=30 Nov 04 01:32:41.461708 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:41.459127 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:41.740669 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:41.735335 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:42.234588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:42.234492 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:42.551590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:42.544683 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:43.197552 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:43.197517 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:43.385621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:43.385424 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:43.679590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:43.678808 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:43.923583 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:43.922777 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j"] Nov 04 01:32:43.923583 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:43.923371 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" podUID="1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" containerName="istio-proxy" containerID="cri-o://49d908083b7dcf8a18fe7782318c86de1b2921b47748c0d4edaff508698539f6" gracePeriod=30 Nov 04 01:32:44.130921 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:44.105338 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:44.565585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:44.555684 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:45.740661 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:45.740613 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:46.407615 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.407545 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:46.742180 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.742156 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:46.761132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.761107 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-token\" (UniqueName: \"kubernetes.io/projected/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-token\") pod \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " Nov 04 01:32:46.761252 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.761211 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-data\") pod \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " Nov 04 01:32:46.761296 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.761263 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-credential-socket\") pod \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " Nov 04 01:32:46.761536 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.761506 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-data" pod "dbba8e43-9895-4f2e-b1a1-a91208543bf1" (UID: "dbba8e43-9895-4f2e-b1a1-a91208543bf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:46.761536 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.761525 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-workload-socket\") pod \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " Nov 04 01:32:46.761868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.761542 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-credential-socket" pod "dbba8e43-9895-4f2e-b1a1-a91208543bf1" (UID: "dbba8e43-9895-4f2e-b1a1-a91208543bf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:46.761868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.761605 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-envoy\") pod \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " Nov 04 01:32:46.761868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.761628 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-podinfo\") pod \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " Nov 04 01:32:46.761868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.761653 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-workload-certs\") pod \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " Nov 04 01:32:46.761868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.761746 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-workload-socket" pod "dbba8e43-9895-4f2e-b1a1-a91208543bf1" (UID: "dbba8e43-9895-4f2e-b1a1-a91208543bf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:46.761868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.761796 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istiod-ca-cert\") pod \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " Nov 04 01:32:46.761868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.761837 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbba8e43-9895-4f2e-b1a1-a91208543bf1-kube-api-access-kjgd5\" (UniqueName: \"kubernetes.io/projected/dbba8e43-9895-4f2e-b1a1-a91208543bf1-kube-api-access-kjgd5\") pod \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\" (UID: \"dbba8e43-9895-4f2e-b1a1-a91208543bf1\") " Nov 04 01:32:46.761868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.761885 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-workload-certs" pod "dbba8e43-9895-4f2e-b1a1-a91208543bf1" (UID: "dbba8e43-9895-4f2e-b1a1-a91208543bf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:46.762356 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.762120 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istiod-ca-cert" pod "dbba8e43-9895-4f2e-b1a1-a91208543bf1" (UID: "dbba8e43-9895-4f2e-b1a1-a91208543bf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:46.762636 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.762617 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:46.762636 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.762636 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:46.762774 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.762645 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:46.762774 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.762657 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:46.762774 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.762665 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:46.763718 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.763662 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-token" pod "dbba8e43-9895-4f2e-b1a1-a91208543bf1" (UID: "dbba8e43-9895-4f2e-b1a1-a91208543bf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:46.764623 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.764369 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-podinfo" pod "dbba8e43-9895-4f2e-b1a1-a91208543bf1" (UID: "dbba8e43-9895-4f2e-b1a1-a91208543bf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:32:46.764796 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.764768 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbba8e43-9895-4f2e-b1a1-a91208543bf1-kube-api-access-kjgd5" pod "dbba8e43-9895-4f2e-b1a1-a91208543bf1" (UID: "dbba8e43-9895-4f2e-b1a1-a91208543bf1"). InnerVolumeSpecName "kube-api-access-kjgd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:46.764889 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.764823 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-envoy" pod "dbba8e43-9895-4f2e-b1a1-a91208543bf1" (UID: "dbba8e43-9895-4f2e-b1a1-a91208543bf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:46.864616 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.864512 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:46.864616 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.864538 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:46.864616 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.864597 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjgd5\" (UniqueName: \"kubernetes.io/projected/dbba8e43-9895-4f2e-b1a1-a91208543bf1-kube-api-access-kjgd5\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:46.864616 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:46.864614 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbba8e43-9895-4f2e-b1a1-a91208543bf1-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:47.277624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:47.277555 4852 generic.go:356] "Generic (PLEG): container finished" podID="dbba8e43-9895-4f2e-b1a1-a91208543bf1" containerID="680e02a4ad15ee2e3f3a7cefc134176ab63ef5d6e4434bc7dbf3c8d801d7a194" exitCode=0 Nov 04 01:32:47.277624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:47.277622 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" Nov 04 01:32:47.277860 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:47.277657 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" event={"ID":"dbba8e43-9895-4f2e-b1a1-a91208543bf1","Type":"ContainerDied","Data":"680e02a4ad15ee2e3f3a7cefc134176ab63ef5d6e4434bc7dbf3c8d801d7a194"} Nov 04 01:32:47.277860 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:47.277712 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl" event={"ID":"dbba8e43-9895-4f2e-b1a1-a91208543bf1","Type":"ContainerDied","Data":"3aba1d1e6bb568590a31eede0c830a5b0353106c985832072340f3f8c9bc0581"} Nov 04 01:32:47.277860 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:47.277734 4852 scope.go:122] "RemoveContainer" containerID="680e02a4ad15ee2e3f3a7cefc134176ab63ef5d6e4434bc7dbf3c8d801d7a194" Nov 04 01:32:47.289698 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:47.289683 4852 scope.go:122] "RemoveContainer" containerID="680e02a4ad15ee2e3f3a7cefc134176ab63ef5d6e4434bc7dbf3c8d801d7a194" Nov 04 01:32:47.289991 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:32:47.289967 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"680e02a4ad15ee2e3f3a7cefc134176ab63ef5d6e4434bc7dbf3c8d801d7a194\": container with ID starting with 680e02a4ad15ee2e3f3a7cefc134176ab63ef5d6e4434bc7dbf3c8d801d7a194 not found: ID does not exist" containerID="680e02a4ad15ee2e3f3a7cefc134176ab63ef5d6e4434bc7dbf3c8d801d7a194" Nov 04 01:32:47.290106 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:47.289997 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"680e02a4ad15ee2e3f3a7cefc134176ab63ef5d6e4434bc7dbf3c8d801d7a194"} err="failed to get container status \"680e02a4ad15ee2e3f3a7cefc134176ab63ef5d6e4434bc7dbf3c8d801d7a194\": rpc error: code = NotFound desc = could not find container \"680e02a4ad15ee2e3f3a7cefc134176ab63ef5d6e4434bc7dbf3c8d801d7a194\": container with ID starting with 680e02a4ad15ee2e3f3a7cefc134176ab63ef5d6e4434bc7dbf3c8d801d7a194 not found: ID does not exist" Nov 04 01:32:47.290744 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:47.290724 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:47.411669 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:47.411635 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl"] Nov 04 01:32:47.424649 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:47.415094 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-c6whl"] Nov 04 01:32:47.997680 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:47.997648 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbba8e43-9895-4f2e-b1a1-a91208543bf1" path="/var/lib/kubelet/pods/dbba8e43-9895-4f2e-b1a1-a91208543bf1/volumes" Nov 04 01:32:48.187583 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:48.179313 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:48.622646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:48.622614 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:48.810589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:48.808130 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:49.268198 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.268157 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:49.323737 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.323649 4852 generic.go:356] "Generic (PLEG): container finished" podID="1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" containerID="49d908083b7dcf8a18fe7782318c86de1b2921b47748c0d4edaff508698539f6" exitCode=0 Nov 04 01:32:49.324213 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.324188 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" event={"ID":"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a","Type":"ContainerDied","Data":"49d908083b7dcf8a18fe7782318c86de1b2921b47748c0d4edaff508698539f6"} Nov 04 01:32:49.324471 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.324452 4852 scope.go:122] "RemoveContainer" containerID="7373d1b8c9784e142253cb4d2cb5ab006a4159586628f7ea493d25eae8604db7" Nov 04 01:32:49.535165 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.535125 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:49.569378 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.569349 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:32:49.607111 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.607038 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istiod-ca-cert\") pod \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " Nov 04 01:32:49.607111 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.607071 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-workload-certs\") pod \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " Nov 04 01:32:49.607111 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.607095 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-envoy\") pod \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " Nov 04 01:32:49.607111 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.607112 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-data\") pod \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " Nov 04 01:32:49.607434 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.607170 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-kube-api-access-rvp9q\" (UniqueName: \"kubernetes.io/projected/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-kube-api-access-rvp9q\") pod \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " Nov 04 01:32:49.607434 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.607191 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-credential-socket\") pod \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " Nov 04 01:32:49.607434 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.607220 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-podinfo\") pod \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " Nov 04 01:32:49.607434 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.607338 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-token\" (UniqueName: \"kubernetes.io/projected/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-token\") pod \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " Nov 04 01:32:49.607434 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.607384 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-workload-certs" pod "1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" (UID: "1858cc03-ee2a-4e57-8d4e-7a10f6194d1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:49.607434 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.607403 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-workload-socket\") pod \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\" (UID: \"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a\") " Nov 04 01:32:49.607434 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.607426 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-data" pod "1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" (UID: "1858cc03-ee2a-4e57-8d4e-7a10f6194d1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:49.607808 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.607617 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-workload-socket" pod "1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" (UID: "1858cc03-ee2a-4e57-8d4e-7a10f6194d1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:49.607808 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.607616 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istiod-ca-cert" pod "1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" (UID: "1858cc03-ee2a-4e57-8d4e-7a10f6194d1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:49.607912 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.607800 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-credential-socket" pod "1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" (UID: "1858cc03-ee2a-4e57-8d4e-7a10f6194d1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:49.608444 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.608404 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:49.608444 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.608435 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:49.608641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.608451 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:49.608641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.608465 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:49.608641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.608479 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:49.609948 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.609923 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-token" pod "1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" (UID: "1858cc03-ee2a-4e57-8d4e-7a10f6194d1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:49.610294 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.610263 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-podinfo" pod "1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" (UID: "1858cc03-ee2a-4e57-8d4e-7a10f6194d1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:32:49.610409 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.610308 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-kube-api-access-rvp9q" pod "1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" (UID: "1858cc03-ee2a-4e57-8d4e-7a10f6194d1a"). InnerVolumeSpecName "kube-api-access-rvp9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:49.614782 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.614754 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-envoy" pod "1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" (UID: "1858cc03-ee2a-4e57-8d4e-7a10f6194d1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:49.709506 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.709478 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:49.709506 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.709506 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvp9q\" (UniqueName: \"kubernetes.io/projected/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-kube-api-access-rvp9q\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:49.709744 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.709517 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:49.709744 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:49.709527 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.309640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.295876 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb"] Nov 04 01:32:50.314851 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.314809 4852 cpu_manager.go:414] "containerMap: removing container" podUID="dbba8e43-9895-4f2e-b1a1-a91208543bf1" containerName="istio-proxy" Nov 04 01:32:50.315025 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.315010 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbba8e43-9895-4f2e-b1a1-a91208543bf1" containerName="istio-proxy" Nov 04 01:32:50.315183 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.315170 4852 cpu_manager.go:414] "containerMap: removing container" podUID="1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" containerName="istio-proxy" Nov 04 01:32:50.315274 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.315262 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" containerName="istio-proxy" Nov 04 01:32:50.315388 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.315377 4852 cpu_manager.go:414] "containerMap: removing container" podUID="4b90a88e-68fc-4841-b289-5af7080408d6" containerName="maas-api" Nov 04 01:32:50.315493 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.315482 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b90a88e-68fc-4841-b289-5af7080408d6" containerName="maas-api" Nov 04 01:32:50.320141 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.320119 4852 cpu_manager.go:414] "containerMap: removing container" podUID="1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" containerName="istio-proxy" Nov 04 01:32:50.330291 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.330266 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" containerName="istio-proxy" Nov 04 01:32:50.340265 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.330701 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbba8e43-9895-4f2e-b1a1-a91208543bf1" containerName="istio-proxy" Nov 04 01:32:50.340474 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.340455 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" containerName="istio-proxy" Nov 04 01:32:50.340612 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.340600 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b90a88e-68fc-4841-b289-5af7080408d6" containerName="maas-api" Nov 04 01:32:50.356661 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.356632 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" containerName="istio-proxy" Nov 04 01:32:50.462052 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.434081 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:50.505828 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.503842 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb"] Nov 04 01:32:50.505828 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.503944 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" Nov 04 01:32:50.508963 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.508900 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" Nov 04 01:32:50.509692 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.508904 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j" event={"ID":"1858cc03-ee2a-4e57-8d4e-7a10f6194d1a","Type":"ContainerDied","Data":"a8ad8d36c60bd7b3dc086249553323905dcc28e0e9b5774ca1eadc5897ad17c7"} Nov 04 01:32:50.509692 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.509063 4852 scope.go:122] "RemoveContainer" containerID="49d908083b7dcf8a18fe7782318c86de1b2921b47748c0d4edaff508698539f6" Nov 04 01:32:50.516422 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.516399 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8e9791-serving-cert\"" Nov 04 01:32:50.517723 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.517699 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d5e42b9-cad8-4145-8824-f972cc7d6505-metrics-certs\") pod \"maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb\" (UID: \"7d5e42b9-cad8-4145-8824-f972cc7d6505\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" Nov 04 01:32:50.517833 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.517815 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxgwc\" (UniqueName: \"kubernetes.io/projected/7d5e42b9-cad8-4145-8824-f972cc7d6505-kube-api-access-gxgwc\") pod \"maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb\" (UID: \"7d5e42b9-cad8-4145-8824-f972cc7d6505\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" Nov 04 01:32:50.517953 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.517939 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d5e42b9-cad8-4145-8824-f972cc7d6505-openshift-service-ca\") pod \"maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb\" (UID: \"7d5e42b9-cad8-4145-8824-f972cc7d6505\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" Nov 04 01:32:50.518089 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.518041 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d5e42b9-cad8-4145-8824-f972cc7d6505-maas-api-tls\") pod \"maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb\" (UID: \"7d5e42b9-cad8-4145-8824-f972cc7d6505\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" Nov 04 01:32:50.566258 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.566180 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j"] Nov 04 01:32:50.574917 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.574893 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-cqd6j"] Nov 04 01:32:50.653484 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.653454 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:50.655069 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.655041 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d5e42b9-cad8-4145-8824-f972cc7d6505-metrics-certs\") pod \"maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb\" (UID: \"7d5e42b9-cad8-4145-8824-f972cc7d6505\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" Nov 04 01:32:50.689836 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.689744 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxgwc\" (UniqueName: \"kubernetes.io/projected/7d5e42b9-cad8-4145-8824-f972cc7d6505-kube-api-access-gxgwc\") pod \"maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb\" (UID: \"7d5e42b9-cad8-4145-8824-f972cc7d6505\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" Nov 04 01:32:50.690030 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.689889 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d5e42b9-cad8-4145-8824-f972cc7d6505-openshift-service-ca\") pod \"maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb\" (UID: \"7d5e42b9-cad8-4145-8824-f972cc7d6505\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" Nov 04 01:32:50.690100 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.690046 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d5e42b9-cad8-4145-8824-f972cc7d6505-maas-api-tls\") pod \"maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb\" (UID: \"7d5e42b9-cad8-4145-8824-f972cc7d6505\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" Nov 04 01:32:50.695585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.691282 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d5e42b9-cad8-4145-8824-f972cc7d6505-openshift-service-ca\") pod \"maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb\" (UID: \"7d5e42b9-cad8-4145-8824-f972cc7d6505\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" Nov 04 01:32:50.734620 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.734189 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxgwc\" (UniqueName: \"kubernetes.io/projected/7d5e42b9-cad8-4145-8824-f972cc7d6505-kube-api-access-gxgwc\") pod \"maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb\" (UID: \"7d5e42b9-cad8-4145-8824-f972cc7d6505\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" Nov 04 01:32:50.736028 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.736001 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d5e42b9-cad8-4145-8824-f972cc7d6505-metrics-certs\") pod \"maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb\" (UID: \"7d5e42b9-cad8-4145-8824-f972cc7d6505\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" Nov 04 01:32:50.736155 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.736056 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d5e42b9-cad8-4145-8824-f972cc7d6505-maas-api-tls\") pod \"maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb\" (UID: \"7d5e42b9-cad8-4145-8824-f972cc7d6505\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" Nov 04 01:32:50.824404 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.824326 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:50.825311 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:50.825288 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" Nov 04 01:32:51.391446 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:51.391415 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb"] Nov 04 01:32:51.527793 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:51.527751 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" event={"ID":"7d5e42b9-cad8-4145-8824-f972cc7d6505","Type":"ContainerStarted","Data":"1ac7693dfed7e7d15c1b161ae0875e5dde7e86e0f3d96b2f2d1b64ccbc98f769"} Nov 04 01:32:51.581358 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:51.581329 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:52.012027 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.010282 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1858cc03-ee2a-4e57-8d4e-7a10f6194d1a" path="/var/lib/kubelet/pods/1858cc03-ee2a-4e57-8d4e-7a10f6194d1a/volumes" Nov 04 01:32:52.251264 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.251237 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:52.336763 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.336727 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-workload-certs\") pod \"f244db1e-5289-4885-9e1c-2b63731821ca\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " Nov 04 01:32:52.337021 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.336960 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f244db1e-5289-4885-9e1c-2b63731821ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f244db1e-5289-4885-9e1c-2b63731821ca-istio-podinfo\") pod \"f244db1e-5289-4885-9e1c-2b63731821ca\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " Nov 04 01:32:52.337163 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.337059 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-workload-certs" pod "f244db1e-5289-4885-9e1c-2b63731821ca" (UID: "f244db1e-5289-4885-9e1c-2b63731821ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:52.337163 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.337079 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f244db1e-5289-4885-9e1c-2b63731821ca-kube-api-access-bblfx\" (UniqueName: \"kubernetes.io/projected/f244db1e-5289-4885-9e1c-2b63731821ca-kube-api-access-bblfx\") pod \"f244db1e-5289-4885-9e1c-2b63731821ca\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " Nov 04 01:32:52.337163 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.337154 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-istio-envoy\") pod \"f244db1e-5289-4885-9e1c-2b63731821ca\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " Nov 04 01:32:52.337357 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.337186 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f244db1e-5289-4885-9e1c-2b63731821ca-istio-token\" (UniqueName: \"kubernetes.io/projected/f244db1e-5289-4885-9e1c-2b63731821ca-istio-token\") pod \"f244db1e-5289-4885-9e1c-2b63731821ca\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " Nov 04 01:32:52.337357 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.337238 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-workload-socket\") pod \"f244db1e-5289-4885-9e1c-2b63731821ca\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " Nov 04 01:32:52.337357 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.337285 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-credential-socket\") pod \"f244db1e-5289-4885-9e1c-2b63731821ca\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " Nov 04 01:32:52.337357 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.337320 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f244db1e-5289-4885-9e1c-2b63731821ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f244db1e-5289-4885-9e1c-2b63731821ca-istiod-ca-cert\") pod \"f244db1e-5289-4885-9e1c-2b63731821ca\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " Nov 04 01:32:52.337611 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.337505 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-istio-data\") pod \"f244db1e-5289-4885-9e1c-2b63731821ca\" (UID: \"f244db1e-5289-4885-9e1c-2b63731821ca\") " Nov 04 01:32:52.338759 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.338220 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f244db1e-5289-4885-9e1c-2b63731821ca-istiod-ca-cert" pod "f244db1e-5289-4885-9e1c-2b63731821ca" (UID: "f244db1e-5289-4885-9e1c-2b63731821ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:52.338759 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.338450 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-credential-socket" pod "f244db1e-5289-4885-9e1c-2b63731821ca" (UID: "f244db1e-5289-4885-9e1c-2b63731821ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:52.338759 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.338503 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:52.338759 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.338528 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f244db1e-5289-4885-9e1c-2b63731821ca-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:52.338759 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.338671 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-istio-data" pod "f244db1e-5289-4885-9e1c-2b63731821ca" (UID: "f244db1e-5289-4885-9e1c-2b63731821ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:52.338759 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.338725 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-workload-socket" pod "f244db1e-5289-4885-9e1c-2b63731821ca" (UID: "f244db1e-5289-4885-9e1c-2b63731821ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:52.340012 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.339987 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f244db1e-5289-4885-9e1c-2b63731821ca-istio-token" pod "f244db1e-5289-4885-9e1c-2b63731821ca" (UID: "f244db1e-5289-4885-9e1c-2b63731821ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:52.340116 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.340027 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f244db1e-5289-4885-9e1c-2b63731821ca-kube-api-access-bblfx" pod "f244db1e-5289-4885-9e1c-2b63731821ca" (UID: "f244db1e-5289-4885-9e1c-2b63731821ca"). InnerVolumeSpecName "kube-api-access-bblfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:52.340483 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.340454 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f244db1e-5289-4885-9e1c-2b63731821ca-istio-podinfo" pod "f244db1e-5289-4885-9e1c-2b63731821ca" (UID: "f244db1e-5289-4885-9e1c-2b63731821ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:32:52.341396 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.341371 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-istio-envoy" pod "f244db1e-5289-4885-9e1c-2b63731821ca" (UID: "f244db1e-5289-4885-9e1c-2b63731821ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:52.440341 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.440309 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:52.440341 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.440338 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f244db1e-5289-4885-9e1c-2b63731821ca-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:52.440723 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.440348 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:52.440723 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.440358 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:52.440723 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.440366 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f244db1e-5289-4885-9e1c-2b63731821ca-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:52.440723 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.440374 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f244db1e-5289-4885-9e1c-2b63731821ca-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:52.440723 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.440383 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bblfx\" (UniqueName: \"kubernetes.io/projected/f244db1e-5289-4885-9e1c-2b63731821ca-kube-api-access-bblfx\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:52.541320 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.541222 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" event={"ID":"7d5e42b9-cad8-4145-8824-f972cc7d6505","Type":"ContainerStarted","Data":"70f4abf51b789c0b945c613e1d490dd3d5d957b6108f7e58d52afecec81b13e0"} Nov 04 01:32:52.541320 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.541299 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" Nov 04 01:32:52.543033 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.543003 4852 generic.go:356] "Generic (PLEG): container finished" podID="f244db1e-5289-4885-9e1c-2b63731821ca" containerID="b165ff6f0416d8ecf3a958cf04cd7c4a8ccff0226ef2d3b6024fadb83f510451" exitCode=0 Nov 04 01:32:52.543178 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.543036 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" event={"ID":"f244db1e-5289-4885-9e1c-2b63731821ca","Type":"ContainerDied","Data":"b165ff6f0416d8ecf3a958cf04cd7c4a8ccff0226ef2d3b6024fadb83f510451"} Nov 04 01:32:52.543178 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.543067 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" Nov 04 01:32:52.543178 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.543084 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc" event={"ID":"f244db1e-5289-4885-9e1c-2b63731821ca","Type":"ContainerDied","Data":"0e2aef6de0602b5beed56efefc7eaf0d85e6610a352325da93918c42825c5754"} Nov 04 01:32:52.543178 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.543106 4852 scope.go:122] "RemoveContainer" containerID="b165ff6f0416d8ecf3a958cf04cd7c4a8ccff0226ef2d3b6024fadb83f510451" Nov 04 01:32:52.556010 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.555992 4852 scope.go:122] "RemoveContainer" containerID="b165ff6f0416d8ecf3a958cf04cd7c4a8ccff0226ef2d3b6024fadb83f510451" Nov 04 01:32:52.556282 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:32:52.556258 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b165ff6f0416d8ecf3a958cf04cd7c4a8ccff0226ef2d3b6024fadb83f510451\": container with ID starting with b165ff6f0416d8ecf3a958cf04cd7c4a8ccff0226ef2d3b6024fadb83f510451 not found: ID does not exist" containerID="b165ff6f0416d8ecf3a958cf04cd7c4a8ccff0226ef2d3b6024fadb83f510451" Nov 04 01:32:52.556350 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.556297 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b165ff6f0416d8ecf3a958cf04cd7c4a8ccff0226ef2d3b6024fadb83f510451"} err="failed to get container status \"b165ff6f0416d8ecf3a958cf04cd7c4a8ccff0226ef2d3b6024fadb83f510451\": rpc error: code = NotFound desc = could not find container \"b165ff6f0416d8ecf3a958cf04cd7c4a8ccff0226ef2d3b6024fadb83f510451\": container with ID starting with b165ff6f0416d8ecf3a958cf04cd7c4a8ccff0226ef2d3b6024fadb83f510451 not found: ID does not exist" Nov 04 01:32:52.573469 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.573428 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" podStartSLOduration=1.813144272 podStartE2EDuration="2.573412288s" totalImagesPullingTime="760.268016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:52.568137595 +0000 UTC m=+2054.683247716" watchObservedRunningTime="2026-11-04 01:32:52.573412288 +0000 UTC m=+2054.688522407" Nov 04 01:32:52.590110 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.590087 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc"] Nov 04 01:32:52.598994 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.598974 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-q85lc"] Nov 04 01:32:52.724772 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:52.724647 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:53.560061 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:53.560029 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:53.997328 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:53.997241 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f244db1e-5289-4885-9e1c-2b63731821ca" path="/var/lib/kubelet/pods/f244db1e-5289-4885-9e1c-2b63731821ca/volumes" Nov 04 01:32:54.141665 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:54.141613 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:54.310592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:54.307930 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:54.956589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:54.945809 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:55.324881 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:55.324854 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:55.669526 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:55.660538 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:56.274626 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:56.274581 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:56.434630 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:56.432087 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:57.070790 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:57.070755 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:57.687529 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:57.687500 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:58.467405 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:58.467306 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:32:58.609815 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:58.609788 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" Nov 04 01:32:59.135708 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:32:59.135678 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:00.084280 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:33:00.084223 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:33:00.211315 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:00.206556 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:00.961641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:00.961602 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:01.177045 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:01.177015 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:01.367119 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:01.367083 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:01.559536 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:01.559505 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:02.447735 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:02.447699 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:03.320632 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:03.319906 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:03.572419 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:03.572329 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:04.194063 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:04.193928 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:04.346657 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:04.346621 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:04.893186 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:04.893156 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:05.065842 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:05.065788 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:05.579267 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:05.579230 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:05.849880 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:05.849776 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:06.816591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:06.796094 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:07.909111 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:07.908386 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:08.136284 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:08.136258 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:08.849720 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:08.849688 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:09.565632 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:09.565294 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:09.886298 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:09.886209 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:10.329555 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:10.329518 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:10.469463 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:10.469434 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:11.019864 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:11.019836 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:11.815673 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:11.815638 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:11.972421 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:11.972385 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:12.410279 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:12.410248 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:12.738504 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:12.738350 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:13.731152 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:13.731119 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:14.637140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:14.637104 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:15.101433 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:15.101383 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:15.367418 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:15.367340 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:15.568335 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:15.568305 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:15.758101 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:15.753254 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:16.065587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:16.065475 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4"] Nov 04 01:33:16.066120 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:16.066089 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" podUID="2af6084e-6b1b-4728-b4d0-6ca48db7a30b" containerName="istio-proxy" containerID="cri-o://ab6f6b48bcc284ffd00135ad9b5508d9f3ac4400cb370d99c9536c2ee972854a" gracePeriod=30 Nov 04 01:33:16.095250 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:16.095222 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc"] Nov 04 01:33:16.095428 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:16.095410 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" podUID="784fa5a1-0f23-4a6e-a6c4-eae42e88f7da" containerName="istio-proxy" containerID="cri-o://005f8df3f988624bff36a057c00e0df6e8c3181b72b3efe39c1913e008180752" gracePeriod=30 Nov 04 01:33:16.225588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:16.214416 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:16.272080 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:16.272046 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:33:16.272350 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:16.272326 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" podUID="2af6084e-6b1b-4728-b4d0-6ca48db7a30b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:33:16.563589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:16.537421 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:16.586597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:16.586143 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:33:16.586597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:16.586262 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" podUID="784fa5a1-0f23-4a6e-a6c4-eae42e88f7da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:33:17.178457 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:17.178286 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:17.525788 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:17.525757 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:18.072264 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:18.072229 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:18.990500 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:18.988188 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:19.854360 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:19.854327 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:20.130349 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:20.129153 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:20.359873 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:20.359841 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:20.623293 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:20.623244 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:20.813031 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:20.812995 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:21.120643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:21.113695 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:21.418636 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:21.409535 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:21.878827 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:21.878774 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:22.000320 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.000266 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:33:22.057254 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.057227 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:33:22.164101 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.164011 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-podinfo\") pod \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " Nov 04 01:33:22.164101 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.164052 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-data\") pod \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " Nov 04 01:33:22.164101 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.164100 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-envoy\") pod \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " Nov 04 01:33:22.164400 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.164119 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-workload-socket\") pod \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " Nov 04 01:33:22.164400 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.164145 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-token\" (UniqueName: \"kubernetes.io/projected/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-token\") pod \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " Nov 04 01:33:22.164400 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.164201 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-kube-api-access-fn9z8\" (UniqueName: \"kubernetes.io/projected/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-kube-api-access-fn9z8\") pod \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " Nov 04 01:33:22.164400 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.164246 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-podinfo\") pod \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " Nov 04 01:33:22.164400 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.164276 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-data\") pod \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " Nov 04 01:33:22.164400 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.164330 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istiod-ca-cert\") pod \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " Nov 04 01:33:22.164761 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.164406 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istiod-ca-cert\") pod \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " Nov 04 01:33:22.164761 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.164453 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-credential-socket\") pod \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " Nov 04 01:33:22.164761 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.164496 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-data" pod "2af6084e-6b1b-4728-b4d0-6ca48db7a30b" (UID: "2af6084e-6b1b-4728-b4d0-6ca48db7a30b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.164761 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.164738 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-workload-socket" pod "784fa5a1-0f23-4a6e-a6c4-eae42e88f7da" (UID: "784fa5a1-0f23-4a6e-a6c4-eae42e88f7da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.164994 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.164967 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-data" pod "784fa5a1-0f23-4a6e-a6c4-eae42e88f7da" (UID: "784fa5a1-0f23-4a6e-a6c4-eae42e88f7da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.165061 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.165029 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istiod-ca-cert" pod "784fa5a1-0f23-4a6e-a6c4-eae42e88f7da" (UID: "784fa5a1-0f23-4a6e-a6c4-eae42e88f7da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:22.165486 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.165446 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istiod-ca-cert" pod "2af6084e-6b1b-4728-b4d0-6ca48db7a30b" (UID: "2af6084e-6b1b-4728-b4d0-6ca48db7a30b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:22.168012 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.167985 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-credential-socket" pod "784fa5a1-0f23-4a6e-a6c4-eae42e88f7da" (UID: "784fa5a1-0f23-4a6e-a6c4-eae42e88f7da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.168155 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.168105 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-kube-api-access-kgvb6\" (UniqueName: \"kubernetes.io/projected/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-kube-api-access-kgvb6\") pod \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " Nov 04 01:33:22.169582 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.169164 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-workload-certs\") pod \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " Nov 04 01:33:22.169582 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.169271 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-workload-socket\") pod \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " Nov 04 01:33:22.169582 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.169347 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-workload-certs\") pod \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " Nov 04 01:33:22.169582 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.169378 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-token\" (UniqueName: \"kubernetes.io/projected/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-token\") pod \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\" (UID: \"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da\") " Nov 04 01:33:22.170479 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.170237 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-workload-socket" pod "2af6084e-6b1b-4728-b4d0-6ca48db7a30b" (UID: "2af6084e-6b1b-4728-b4d0-6ca48db7a30b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.170479 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.170389 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-workload-certs" pod "784fa5a1-0f23-4a6e-a6c4-eae42e88f7da" (UID: "784fa5a1-0f23-4a6e-a6c4-eae42e88f7da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.170479 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.170438 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-workload-certs" pod "2af6084e-6b1b-4728-b4d0-6ca48db7a30b" (UID: "2af6084e-6b1b-4728-b4d0-6ca48db7a30b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.171044 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.171027 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-envoy\") pod \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " Nov 04 01:33:22.171260 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.171244 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-credential-socket\") pod \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\" (UID: \"2af6084e-6b1b-4728-b4d0-6ca48db7a30b\") " Nov 04 01:33:22.171532 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.171500 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-credential-socket" pod "2af6084e-6b1b-4728-b4d0-6ca48db7a30b" (UID: "2af6084e-6b1b-4728-b4d0-6ca48db7a30b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.172253 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.172230 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.172377 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.172364 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.172478 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.172465 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.173201 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.173184 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.173293 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.172748 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-envoy" pod "784fa5a1-0f23-4a6e-a6c4-eae42e88f7da" (UID: "784fa5a1-0f23-4a6e-a6c4-eae42e88f7da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.173293 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.173172 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-token" pod "2af6084e-6b1b-4728-b4d0-6ca48db7a30b" (UID: "2af6084e-6b1b-4728-b4d0-6ca48db7a30b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:22.173654 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.173236 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.173654 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.173317 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.173654 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.173330 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.173654 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.173344 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.173654 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.173358 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.173654 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.173388 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.175167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.175136 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-kube-api-access-fn9z8" pod "2af6084e-6b1b-4728-b4d0-6ca48db7a30b" (UID: "2af6084e-6b1b-4728-b4d0-6ca48db7a30b"). InnerVolumeSpecName "kube-api-access-fn9z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:22.175349 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.175327 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-podinfo" pod "2af6084e-6b1b-4728-b4d0-6ca48db7a30b" (UID: "2af6084e-6b1b-4728-b4d0-6ca48db7a30b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:22.175449 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.175425 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-podinfo" pod "784fa5a1-0f23-4a6e-a6c4-eae42e88f7da" (UID: "784fa5a1-0f23-4a6e-a6c4-eae42e88f7da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:22.176628 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.176537 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-envoy" pod "2af6084e-6b1b-4728-b4d0-6ca48db7a30b" (UID: "2af6084e-6b1b-4728-b4d0-6ca48db7a30b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.176808 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.176789 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-token" pod "784fa5a1-0f23-4a6e-a6c4-eae42e88f7da" (UID: "784fa5a1-0f23-4a6e-a6c4-eae42e88f7da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:22.177403 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.177388 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-kube-api-access-kgvb6" pod "784fa5a1-0f23-4a6e-a6c4-eae42e88f7da" (UID: "784fa5a1-0f23-4a6e-a6c4-eae42e88f7da"). InnerVolumeSpecName "kube-api-access-kgvb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:22.274772 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.274746 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.274931 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.274774 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn9z8\" (UniqueName: \"kubernetes.io/projected/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-kube-api-access-fn9z8\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.274931 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.274791 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.274931 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.274800 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgvb6\" (UniqueName: \"kubernetes.io/projected/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-kube-api-access-kgvb6\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.274931 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.274809 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.274931 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.274817 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2af6084e-6b1b-4728-b4d0-6ca48db7a30b-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.274931 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.274826 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.274931 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.274835 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.294054 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.294026 4852 generic.go:356] "Generic (PLEG): container finished" podID="2af6084e-6b1b-4728-b4d0-6ca48db7a30b" containerID="ab6f6b48bcc284ffd00135ad9b5508d9f3ac4400cb370d99c9536c2ee972854a" exitCode=0 Nov 04 01:33:22.294187 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.294098 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" Nov 04 01:33:22.294187 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.294109 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" event={"ID":"2af6084e-6b1b-4728-b4d0-6ca48db7a30b","Type":"ContainerDied","Data":"ab6f6b48bcc284ffd00135ad9b5508d9f3ac4400cb370d99c9536c2ee972854a"} Nov 04 01:33:22.294282 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.294203 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4" event={"ID":"2af6084e-6b1b-4728-b4d0-6ca48db7a30b","Type":"ContainerDied","Data":"09d12b02e9f2a5b5f50d8b7c8271edcd5a7ab151a059fa84342ca24117dc8e10"} Nov 04 01:33:22.294282 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.294227 4852 scope.go:122] "RemoveContainer" containerID="ab6f6b48bcc284ffd00135ad9b5508d9f3ac4400cb370d99c9536c2ee972854a" Nov 04 01:33:22.295797 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.295774 4852 generic.go:356] "Generic (PLEG): container finished" podID="784fa5a1-0f23-4a6e-a6c4-eae42e88f7da" containerID="005f8df3f988624bff36a057c00e0df6e8c3181b72b3efe39c1913e008180752" exitCode=0 Nov 04 01:33:22.295929 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.295842 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" Nov 04 01:33:22.295929 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.295860 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" event={"ID":"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da","Type":"ContainerDied","Data":"005f8df3f988624bff36a057c00e0df6e8c3181b72b3efe39c1913e008180752"} Nov 04 01:33:22.295929 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.295924 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc" event={"ID":"784fa5a1-0f23-4a6e-a6c4-eae42e88f7da","Type":"ContainerDied","Data":"35b0cdf71212dbb1d92997e4764faf68666596b46b55284b0532b74bed6e816d"} Nov 04 01:33:22.305385 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.305273 4852 scope.go:122] "RemoveContainer" containerID="ab6f6b48bcc284ffd00135ad9b5508d9f3ac4400cb370d99c9536c2ee972854a" Nov 04 01:33:22.305636 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:33:22.305614 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab6f6b48bcc284ffd00135ad9b5508d9f3ac4400cb370d99c9536c2ee972854a\": container with ID starting with ab6f6b48bcc284ffd00135ad9b5508d9f3ac4400cb370d99c9536c2ee972854a not found: ID does not exist" containerID="ab6f6b48bcc284ffd00135ad9b5508d9f3ac4400cb370d99c9536c2ee972854a" Nov 04 01:33:22.305731 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.305641 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab6f6b48bcc284ffd00135ad9b5508d9f3ac4400cb370d99c9536c2ee972854a"} err="failed to get container status \"ab6f6b48bcc284ffd00135ad9b5508d9f3ac4400cb370d99c9536c2ee972854a\": rpc error: code = NotFound desc = could not find container \"ab6f6b48bcc284ffd00135ad9b5508d9f3ac4400cb370d99c9536c2ee972854a\": container with ID starting with ab6f6b48bcc284ffd00135ad9b5508d9f3ac4400cb370d99c9536c2ee972854a not found: ID does not exist" Nov 04 01:33:22.305731 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.305658 4852 scope.go:122] "RemoveContainer" containerID="005f8df3f988624bff36a057c00e0df6e8c3181b72b3efe39c1913e008180752" Nov 04 01:33:22.316346 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.316332 4852 scope.go:122] "RemoveContainer" containerID="005f8df3f988624bff36a057c00e0df6e8c3181b72b3efe39c1913e008180752" Nov 04 01:33:22.316614 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:33:22.316597 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"005f8df3f988624bff36a057c00e0df6e8c3181b72b3efe39c1913e008180752\": container with ID starting with 005f8df3f988624bff36a057c00e0df6e8c3181b72b3efe39c1913e008180752 not found: ID does not exist" containerID="005f8df3f988624bff36a057c00e0df6e8c3181b72b3efe39c1913e008180752" Nov 04 01:33:22.316684 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.316622 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"005f8df3f988624bff36a057c00e0df6e8c3181b72b3efe39c1913e008180752"} err="failed to get container status \"005f8df3f988624bff36a057c00e0df6e8c3181b72b3efe39c1913e008180752\": rpc error: code = NotFound desc = could not find container \"005f8df3f988624bff36a057c00e0df6e8c3181b72b3efe39c1913e008180752\": container with ID starting with 005f8df3f988624bff36a057c00e0df6e8c3181b72b3efe39c1913e008180752 not found: ID does not exist" Nov 04 01:33:22.336381 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.336358 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc"] Nov 04 01:33:22.346916 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.346892 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-7z8sc"] Nov 04 01:33:22.367384 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.367356 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4"] Nov 04 01:33:22.377211 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:22.377172 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-jv8q4"] Nov 04 01:33:23.747617 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:23.747584 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:23.968171 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:23.968136 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:24.011082 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:24.011052 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2af6084e-6b1b-4728-b4d0-6ca48db7a30b" path="/var/lib/kubelet/pods/2af6084e-6b1b-4728-b4d0-6ca48db7a30b/volumes" Nov 04 01:33:24.012641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:24.012618 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="784fa5a1-0f23-4a6e-a6c4-eae42e88f7da" path="/var/lib/kubelet/pods/784fa5a1-0f23-4a6e-a6c4-eae42e88f7da/volumes" Nov 04 01:33:24.342637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:24.338645 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:24.969686 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:24.969654 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:33:25.262583 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:25.258540 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:26.117588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:26.117486 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:26.783588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:26.750293 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:27.115658 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:27.113211 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:28.530590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:28.530126 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:28.856656 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:28.808763 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:29.110787 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:28.982353 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m"] Nov 04 01:33:29.110787 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:29.016676 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" podUID="a4cd0527-183f-484b-8dc5-8fc84aa45b1b" containerName="istio-proxy" containerID="cri-o://ddd324c2e3ccaa3c0a89a02b2431d59a01c232f50d5e0649a1cf34ac4d2b1964" gracePeriod=30 Nov 04 01:33:29.110787 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:29.041699 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr"] Nov 04 01:33:29.124090 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:29.124038 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" podUID="93308cb8-8e7f-45c3-a1f6-bbd045723e67" containerName="istio-proxy" containerID="cri-o://66314a9ef0a948f5d32fb90a73843dda7ca8b2eca8ae141e77129c52f4c5be76" gracePeriod=30 Nov 04 01:33:29.294621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:29.293638 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l"] Nov 04 01:33:29.294621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:29.293887 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" podUID="36db7091-781e-4f96-a979-6ad25e0fb53f" containerName="istio-proxy" containerID="cri-o://051f67ebf8605d67633e79cdcc778071025ef0d5567155753fdec4b2613434b0" gracePeriod=30 Nov 04 01:33:29.336597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:29.330464 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:29.375011 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:29.374915 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:29.389455 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:29.382603 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c"] Nov 04 01:33:29.389455 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:29.383342 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" podUID="a247fcaf-f6ae-499c-83d9-acfdfa09cad0" containerName="istio-proxy" containerID="cri-o://7bc34e24d741f58a32e121838c46b49a793759d33225049adffbea76091b4ee2" gracePeriod=30 Nov 04 01:33:29.555923 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:29.542809 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:29.555923 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:29.547815 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:29.630098 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:29.629616 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4"] Nov 04 01:33:29.670734 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:29.652947 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" podUID="82a7dd85-4ad8-4990-8907-cb64e413cceb" containerName="istio-proxy" containerID="cri-o://bd9cbabd2ff2ea49ee861aea08045312dc508f5760614c80ead0293e4dcc7915" gracePeriod=30 Nov 04 01:33:29.723302 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:29.723254 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:29.725152 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:29.725121 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw"] Nov 04 01:33:29.734235 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:29.726153 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" podUID="169d147a-6807-4836-8eca-4b7aa3d33bd5" containerName="istio-proxy" containerID="cri-o://aa8ecffc130835fd35082b1f1ff491a853a2d44bb45b907b64a30b00d6a40562" gracePeriod=30 Nov 04 01:33:29.789618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:29.786800 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:30.005399 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:29.932931 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz"] Nov 04 01:33:30.005399 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:29.933240 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" podUID="6b03d6b4-7003-4705-b3df-64461213d3bc" containerName="istio-proxy" containerID="cri-o://7cf793fb503d17ce8969dad595474c0b451a20e88b8c69024d5c2e7264531b9e" gracePeriod=30 Nov 04 01:33:30.005399 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:29.966772 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:30.084597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:30.084026 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:30.178017 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:33:30.146776 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:33:30.178017 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:30.168870 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:30.391653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:30.386845 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:30.492588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:30.492405 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:30.545590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:30.535058 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:33:30.555737 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:30.555674 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:30.556517 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:30.556492 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:33:30.560063 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:30.557039 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:30.568522 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:30.557075 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:30.568522 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:30.560838 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:33:30.750134 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:30.748650 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:33:30.810784 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:30.786013 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:30.812381 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:30.812355 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:33:31.040523 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:31.040481 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:31.042479 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:31.042455 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:33:31.084776 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:31.084738 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:31.086313 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:31.086291 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:33:31.181589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:31.171659 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:31.181589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:31.172988 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 04 01:33:31.181589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:31.174327 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:33:31.245959 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:31.245934 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:33:31.257593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:31.255177 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:33:31.426321 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:31.426237 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:33:31.437645 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:31.428459 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:31.438176 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:31.429489 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 04 01:33:31.472852 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:31.472309 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:31.572078 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:31.571352 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:31.711118 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:31.711012 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:31.787486 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:31.787459 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:33:31.788594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:31.788551 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:33:31.812103 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:31.812077 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:33:32.105648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:32.105615 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:32.301974 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:32.301939 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:32.761796 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:32.761728 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:32.952814 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:32.952715 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:33.257648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:33.256472 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:33.615995 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:33.615895 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 04 01:33:33.714617 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:33.708998 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:33.742973 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:33.735870 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:33.825371 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:33.822718 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:34.036485 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:34.036228 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:34.043675 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:34.043628 4852 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:34.046414 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:34.045191 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:34.234710 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:34.232554 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:34.239865 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:34.239821 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:34.414698 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:34.413259 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:34.717282 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:34.717200 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb"] Nov 04 01:33:34.717468 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:34.717439 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" podUID="57537b5d-473c-41da-9e67-6c848a66fee7" containerName="istio-proxy" containerID="cri-o://1d6201d9c2ff6e099ef6ac5252c7c828c889c83a42af01358c2e567bb839831e" gracePeriod=30 Nov 04 01:33:34.789352 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:34.787256 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:34.940687 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:34.916290 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 04 01:33:34.940687 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:34.916351 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" podUID="a4cd0527-183f-484b-8dc5-8fc84aa45b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 04 01:33:34.971309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:34.968203 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:33:34.990255 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:34.976446 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": read tcp 10.134.0.2:51956->10.134.0.40:15021: read: connection reset by peer" start-of-body= Nov 04 01:33:34.990255 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:34.976502 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" podUID="82a7dd85-4ad8-4990-8907-cb64e413cceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": read tcp 10.134.0.2:51956->10.134.0.40:15021: read: connection reset by peer" Nov 04 01:33:35.097865 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.051242 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93308cb8-8e7f-45c3-a1f6-bbd045723e67-kube-api-access-mvj2q\" (UniqueName: \"kubernetes.io/projected/93308cb8-8e7f-45c3-a1f6-bbd045723e67-kube-api-access-mvj2q\") pod \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " Nov 04 01:33:35.097865 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.051370 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-workload-socket\") pod \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " Nov 04 01:33:35.097865 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.051432 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-podinfo\") pod \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " Nov 04 01:33:35.097865 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.056695 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-token\" (UniqueName: \"kubernetes.io/projected/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-token\") pod \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " Nov 04 01:33:35.097865 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.056774 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-credential-socket\") pod \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " Nov 04 01:33:35.097865 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.056895 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istiod-ca-cert\") pod \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " Nov 04 01:33:35.097865 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.057011 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-data\") pod \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " Nov 04 01:33:35.097865 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.057113 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-workload-certs\") pod \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " Nov 04 01:33:35.097865 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.057163 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-envoy\") pod \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\" (UID: \"93308cb8-8e7f-45c3-a1f6-bbd045723e67\") " Nov 04 01:33:35.100005 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.099028 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istiod-ca-cert" pod "93308cb8-8e7f-45c3-a1f6-bbd045723e67" (UID: "93308cb8-8e7f-45c3-a1f6-bbd045723e67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:35.100005 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.099296 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-workload-socket" pod "93308cb8-8e7f-45c3-a1f6-bbd045723e67" (UID: "93308cb8-8e7f-45c3-a1f6-bbd045723e67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.111860 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.109757 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-credential-socket" pod "93308cb8-8e7f-45c3-a1f6-bbd045723e67" (UID: "93308cb8-8e7f-45c3-a1f6-bbd045723e67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.111860 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.110754 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-workload-certs" pod "93308cb8-8e7f-45c3-a1f6-bbd045723e67" (UID: "93308cb8-8e7f-45c3-a1f6-bbd045723e67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.116275 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.116113 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-data" pod "93308cb8-8e7f-45c3-a1f6-bbd045723e67" (UID: "93308cb8-8e7f-45c3-a1f6-bbd045723e67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.180129 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.179631 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.180129 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.179663 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.180129 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.179679 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.180129 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.179695 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.180129 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.179709 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.211814 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.201204 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-envoy" pod "93308cb8-8e7f-45c3-a1f6-bbd045723e67" (UID: "93308cb8-8e7f-45c3-a1f6-bbd045723e67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.216508 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.215724 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93308cb8-8e7f-45c3-a1f6-bbd045723e67-kube-api-access-mvj2q" pod "93308cb8-8e7f-45c3-a1f6-bbd045723e67" (UID: "93308cb8-8e7f-45c3-a1f6-bbd045723e67"). InnerVolumeSpecName "kube-api-access-mvj2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:35.216508 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.216149 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-podinfo" pod "93308cb8-8e7f-45c3-a1f6-bbd045723e67" (UID: "93308cb8-8e7f-45c3-a1f6-bbd045723e67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:35.254765 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.246816 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-token" pod "93308cb8-8e7f-45c3-a1f6-bbd045723e67" (UID: "93308cb8-8e7f-45c3-a1f6-bbd045723e67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:35.254765 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.248909 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:35.433605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.299224 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.433605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.299273 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvj2q\" (UniqueName: \"kubernetes.io/projected/93308cb8-8e7f-45c3-a1f6-bbd045723e67-kube-api-access-mvj2q\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.433605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.299290 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.433605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.299305 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93308cb8-8e7f-45c3-a1f6-bbd045723e67-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.512132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.452208 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:35.512132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.487191 4852 generic.go:356] "Generic (PLEG): container finished" podID="6b03d6b4-7003-4705-b3df-64461213d3bc" containerID="7cf793fb503d17ce8969dad595474c0b451a20e88b8c69024d5c2e7264531b9e" exitCode=0 Nov 04 01:33:35.512132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.487297 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" event={"ID":"6b03d6b4-7003-4705-b3df-64461213d3bc","Type":"ContainerDied","Data":"7cf793fb503d17ce8969dad595474c0b451a20e88b8c69024d5c2e7264531b9e"} Nov 04 01:33:35.512132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.489150 4852 generic.go:356] "Generic (PLEG): container finished" podID="93308cb8-8e7f-45c3-a1f6-bbd045723e67" containerID="66314a9ef0a948f5d32fb90a73843dda7ca8b2eca8ae141e77129c52f4c5be76" exitCode=0 Nov 04 01:33:35.512132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.489240 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" event={"ID":"93308cb8-8e7f-45c3-a1f6-bbd045723e67","Type":"ContainerDied","Data":"66314a9ef0a948f5d32fb90a73843dda7ca8b2eca8ae141e77129c52f4c5be76"} Nov 04 01:33:35.512132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.489283 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" event={"ID":"93308cb8-8e7f-45c3-a1f6-bbd045723e67","Type":"ContainerDied","Data":"083dd58c53950cd264db79f91ed6348c436d8843c6c9da1a7aba84bc1e8e09c2"} Nov 04 01:33:35.512132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.489312 4852 scope.go:122] "RemoveContainer" containerID="66314a9ef0a948f5d32fb90a73843dda7ca8b2eca8ae141e77129c52f4c5be76" Nov 04 01:33:35.512132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.489470 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr" Nov 04 01:33:35.512132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.497359 4852 generic.go:356] "Generic (PLEG): container finished" podID="169d147a-6807-4836-8eca-4b7aa3d33bd5" containerID="aa8ecffc130835fd35082b1f1ff491a853a2d44bb45b907b64a30b00d6a40562" exitCode=0 Nov 04 01:33:35.512132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.497449 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" event={"ID":"169d147a-6807-4836-8eca-4b7aa3d33bd5","Type":"ContainerDied","Data":"aa8ecffc130835fd35082b1f1ff491a853a2d44bb45b907b64a30b00d6a40562"} Nov 04 01:33:35.512132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.500751 4852 generic.go:356] "Generic (PLEG): container finished" podID="82a7dd85-4ad8-4990-8907-cb64e413cceb" containerID="bd9cbabd2ff2ea49ee861aea08045312dc508f5760614c80ead0293e4dcc7915" exitCode=0 Nov 04 01:33:35.512132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.500820 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" event={"ID":"82a7dd85-4ad8-4990-8907-cb64e413cceb","Type":"ContainerDied","Data":"bd9cbabd2ff2ea49ee861aea08045312dc508f5760614c80ead0293e4dcc7915"} Nov 04 01:33:35.512132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.506235 4852 generic.go:356] "Generic (PLEG): container finished" podID="36db7091-781e-4f96-a979-6ad25e0fb53f" containerID="051f67ebf8605d67633e79cdcc778071025ef0d5567155753fdec4b2613434b0" exitCode=0 Nov 04 01:33:35.512132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.506379 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" event={"ID":"36db7091-781e-4f96-a979-6ad25e0fb53f","Type":"ContainerDied","Data":"051f67ebf8605d67633e79cdcc778071025ef0d5567155753fdec4b2613434b0"} Nov 04 01:33:35.512132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.508593 4852 generic.go:356] "Generic (PLEG): container finished" podID="a4cd0527-183f-484b-8dc5-8fc84aa45b1b" containerID="ddd324c2e3ccaa3c0a89a02b2431d59a01c232f50d5e0649a1cf34ac4d2b1964" exitCode=0 Nov 04 01:33:35.512132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.508722 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" event={"ID":"a4cd0527-183f-484b-8dc5-8fc84aa45b1b","Type":"ContainerDied","Data":"ddd324c2e3ccaa3c0a89a02b2431d59a01c232f50d5e0649a1cf34ac4d2b1964"} Nov 04 01:33:35.512132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.510359 4852 generic.go:356] "Generic (PLEG): container finished" podID="a247fcaf-f6ae-499c-83d9-acfdfa09cad0" containerID="7bc34e24d741f58a32e121838c46b49a793759d33225049adffbea76091b4ee2" exitCode=0 Nov 04 01:33:35.512132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.510430 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" event={"ID":"a247fcaf-f6ae-499c-83d9-acfdfa09cad0","Type":"ContainerDied","Data":"7bc34e24d741f58a32e121838c46b49a793759d33225049adffbea76091b4ee2"} Nov 04 01:33:35.539907 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.538896 4852 scope.go:122] "RemoveContainer" containerID="66314a9ef0a948f5d32fb90a73843dda7ca8b2eca8ae141e77129c52f4c5be76" Nov 04 01:33:35.539907 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:33:35.539317 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66314a9ef0a948f5d32fb90a73843dda7ca8b2eca8ae141e77129c52f4c5be76\": container with ID starting with 66314a9ef0a948f5d32fb90a73843dda7ca8b2eca8ae141e77129c52f4c5be76 not found: ID does not exist" containerID="66314a9ef0a948f5d32fb90a73843dda7ca8b2eca8ae141e77129c52f4c5be76" Nov 04 01:33:35.539907 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.539351 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66314a9ef0a948f5d32fb90a73843dda7ca8b2eca8ae141e77129c52f4c5be76"} err="failed to get container status \"66314a9ef0a948f5d32fb90a73843dda7ca8b2eca8ae141e77129c52f4c5be76\": rpc error: code = NotFound desc = could not find container \"66314a9ef0a948f5d32fb90a73843dda7ca8b2eca8ae141e77129c52f4c5be76\": container with ID starting with 66314a9ef0a948f5d32fb90a73843dda7ca8b2eca8ae141e77129c52f4c5be76 not found: ID does not exist" Nov 04 01:33:35.545734 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.545708 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:33:35.607822 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.606474 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istiod-ca-cert\") pod \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " Nov 04 01:33:35.607822 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.606584 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-credential-socket\") pod \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " Nov 04 01:33:35.607822 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.606637 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-token\" (UniqueName: \"kubernetes.io/projected/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-token\") pod \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " Nov 04 01:33:35.607822 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.606751 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-kube-api-access-b2lv7\" (UniqueName: \"kubernetes.io/projected/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-kube-api-access-b2lv7\") pod \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " Nov 04 01:33:35.607822 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.606795 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-workload-socket\") pod \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " Nov 04 01:33:35.607822 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.606842 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-envoy\") pod \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " Nov 04 01:33:35.607822 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.606920 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-podinfo\") pod \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " Nov 04 01:33:35.607822 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.606988 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-data\") pod \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " Nov 04 01:33:35.607822 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.607046 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-workload-certs\") pod \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\" (UID: \"a247fcaf-f6ae-499c-83d9-acfdfa09cad0\") " Nov 04 01:33:35.608712 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.608453 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-workload-certs" pod "a247fcaf-f6ae-499c-83d9-acfdfa09cad0" (UID: "a247fcaf-f6ae-499c-83d9-acfdfa09cad0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.609167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.608931 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istiod-ca-cert" pod "a247fcaf-f6ae-499c-83d9-acfdfa09cad0" (UID: "a247fcaf-f6ae-499c-83d9-acfdfa09cad0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:35.610630 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.609371 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-credential-socket" pod "a247fcaf-f6ae-499c-83d9-acfdfa09cad0" (UID: "a247fcaf-f6ae-499c-83d9-acfdfa09cad0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.611205 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.611177 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr"] Nov 04 01:33:35.611284 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.611232 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-567xr"] Nov 04 01:33:35.614215 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.613852 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-workload-socket" pod "a247fcaf-f6ae-499c-83d9-acfdfa09cad0" (UID: "a247fcaf-f6ae-499c-83d9-acfdfa09cad0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.614470 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.614359 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-data" pod "a247fcaf-f6ae-499c-83d9-acfdfa09cad0" (UID: "a247fcaf-f6ae-499c-83d9-acfdfa09cad0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.637352 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.630023 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-kube-api-access-b2lv7" pod "a247fcaf-f6ae-499c-83d9-acfdfa09cad0" (UID: "a247fcaf-f6ae-499c-83d9-acfdfa09cad0"). InnerVolumeSpecName "kube-api-access-b2lv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:35.637352 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.630144 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-envoy" pod "a247fcaf-f6ae-499c-83d9-acfdfa09cad0" (UID: "a247fcaf-f6ae-499c-83d9-acfdfa09cad0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.637352 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.630245 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-token" pod "a247fcaf-f6ae-499c-83d9-acfdfa09cad0" (UID: "a247fcaf-f6ae-499c-83d9-acfdfa09cad0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:35.637352 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.632272 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-podinfo" pod "a247fcaf-f6ae-499c-83d9-acfdfa09cad0" (UID: "a247fcaf-f6ae-499c-83d9-acfdfa09cad0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:35.708620 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.708591 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2lv7\" (UniqueName: \"kubernetes.io/projected/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-kube-api-access-b2lv7\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.708620 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.708617 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.708785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.708627 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.708785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.708636 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.708785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.708645 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.708785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.708654 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.708785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.708663 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.708785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.708672 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.708785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.708680 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a247fcaf-f6ae-499c-83d9-acfdfa09cad0-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.712036 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.712012 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:33:35.809969 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.809943 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-envoy\") pod \"169d147a-6807-4836-8eca-4b7aa3d33bd5\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " Nov 04 01:33:35.810109 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.810027 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/169d147a-6807-4836-8eca-4b7aa3d33bd5-kube-api-access-lzr2z\" (UniqueName: \"kubernetes.io/projected/169d147a-6807-4836-8eca-4b7aa3d33bd5-kube-api-access-lzr2z\") pod \"169d147a-6807-4836-8eca-4b7aa3d33bd5\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " Nov 04 01:33:35.810109 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.810064 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-token\" (UniqueName: \"kubernetes.io/projected/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-token\") pod \"169d147a-6807-4836-8eca-4b7aa3d33bd5\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " Nov 04 01:33:35.810309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.810279 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-podinfo\") pod \"169d147a-6807-4836-8eca-4b7aa3d33bd5\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " Nov 04 01:33:35.810395 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.810329 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-workload-certs\") pod \"169d147a-6807-4836-8eca-4b7aa3d33bd5\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " Nov 04 01:33:35.810502 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.810485 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-data\") pod \"169d147a-6807-4836-8eca-4b7aa3d33bd5\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " Nov 04 01:33:35.810685 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.810661 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/169d147a-6807-4836-8eca-4b7aa3d33bd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169d147a-6807-4836-8eca-4b7aa3d33bd5-istiod-ca-cert\") pod \"169d147a-6807-4836-8eca-4b7aa3d33bd5\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " Nov 04 01:33:35.810769 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.810754 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-credential-socket\") pod \"169d147a-6807-4836-8eca-4b7aa3d33bd5\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " Nov 04 01:33:35.810857 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.810813 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-workload-socket\") pod \"169d147a-6807-4836-8eca-4b7aa3d33bd5\" (UID: \"169d147a-6807-4836-8eca-4b7aa3d33bd5\") " Nov 04 01:33:35.811140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.811011 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-data" pod "169d147a-6807-4836-8eca-4b7aa3d33bd5" (UID: "169d147a-6807-4836-8eca-4b7aa3d33bd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.811488 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.811240 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169d147a-6807-4836-8eca-4b7aa3d33bd5-istiod-ca-cert" pod "169d147a-6807-4836-8eca-4b7aa3d33bd5" (UID: "169d147a-6807-4836-8eca-4b7aa3d33bd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:35.811488 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.811365 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-credential-socket" pod "169d147a-6807-4836-8eca-4b7aa3d33bd5" (UID: "169d147a-6807-4836-8eca-4b7aa3d33bd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.811713 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.811687 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-workload-socket" pod "169d147a-6807-4836-8eca-4b7aa3d33bd5" (UID: "169d147a-6807-4836-8eca-4b7aa3d33bd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.811918 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.811821 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169d147a-6807-4836-8eca-4b7aa3d33bd5-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.811918 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.811844 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.811918 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.811857 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.811918 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.811860 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-workload-certs" pod "169d147a-6807-4836-8eca-4b7aa3d33bd5" (UID: "169d147a-6807-4836-8eca-4b7aa3d33bd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.814555 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.814522 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-token" pod "169d147a-6807-4836-8eca-4b7aa3d33bd5" (UID: "169d147a-6807-4836-8eca-4b7aa3d33bd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:35.814780 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.814756 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-podinfo" pod "169d147a-6807-4836-8eca-4b7aa3d33bd5" (UID: "169d147a-6807-4836-8eca-4b7aa3d33bd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:35.815462 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.815440 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-envoy" pod "169d147a-6807-4836-8eca-4b7aa3d33bd5" (UID: "169d147a-6807-4836-8eca-4b7aa3d33bd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.816169 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.816145 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169d147a-6807-4836-8eca-4b7aa3d33bd5-kube-api-access-lzr2z" pod "169d147a-6807-4836-8eca-4b7aa3d33bd5" (UID: "169d147a-6807-4836-8eca-4b7aa3d33bd5"). InnerVolumeSpecName "kube-api-access-lzr2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:35.913376 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.913334 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzr2z\" (UniqueName: \"kubernetes.io/projected/169d147a-6807-4836-8eca-4b7aa3d33bd5-kube-api-access-lzr2z\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.913376 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.913367 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.913376 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.913380 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.913684 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.913393 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.913684 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.913405 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.913684 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.913417 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169d147a-6807-4836-8eca-4b7aa3d33bd5-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.936817 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.936786 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:33:35.951728 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.951704 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:33:35.955315 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.955295 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:33:35.987868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:35.985378 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:33:36.001217 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.001185 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93308cb8-8e7f-45c3-a1f6-bbd045723e67" path="/var/lib/kubelet/pods/93308cb8-8e7f-45c3-a1f6-bbd045723e67/volumes" Nov 04 01:33:36.013898 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.013866 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-data\") pod \"82a7dd85-4ad8-4990-8907-cb64e413cceb\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " Nov 04 01:33:36.014044 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.013910 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-workload-certs\") pod \"36db7091-781e-4f96-a979-6ad25e0fb53f\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " Nov 04 01:33:36.014044 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.013943 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-data\") pod \"6b03d6b4-7003-4705-b3df-64461213d3bc\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " Nov 04 01:33:36.014044 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.013976 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-podinfo\") pod \"82a7dd85-4ad8-4990-8907-cb64e413cceb\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " Nov 04 01:33:36.014044 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014005 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-envoy\") pod \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " Nov 04 01:33:36.014044 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014038 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-podinfo\") pod \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " Nov 04 01:33:36.014343 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014080 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-credential-socket\") pod \"82a7dd85-4ad8-4990-8907-cb64e413cceb\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " Nov 04 01:33:36.014343 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014120 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-workload-socket\") pod \"36db7091-781e-4f96-a979-6ad25e0fb53f\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " Nov 04 01:33:36.014343 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014156 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b03d6b4-7003-4705-b3df-64461213d3bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b03d6b4-7003-4705-b3df-64461213d3bc-istiod-ca-cert\") pod \"6b03d6b4-7003-4705-b3df-64461213d3bc\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " Nov 04 01:33:36.014343 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014176 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-workload-certs" pod "36db7091-781e-4f96-a979-6ad25e0fb53f" (UID: "36db7091-781e-4f96-a979-6ad25e0fb53f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:36.014343 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014187 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-data" pod "82a7dd85-4ad8-4990-8907-cb64e413cceb" (UID: "82a7dd85-4ad8-4990-8907-cb64e413cceb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:36.014343 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014203 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-workload-certs\") pod \"6b03d6b4-7003-4705-b3df-64461213d3bc\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " Nov 04 01:33:36.014343 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014251 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-envoy\") pod \"36db7091-781e-4f96-a979-6ad25e0fb53f\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " Nov 04 01:33:36.014343 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014286 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-token\" (UniqueName: \"kubernetes.io/projected/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-token\") pod \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " Nov 04 01:33:36.014789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014371 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-data\") pod \"36db7091-781e-4f96-a979-6ad25e0fb53f\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " Nov 04 01:33:36.014789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014408 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-envoy\") pod \"82a7dd85-4ad8-4990-8907-cb64e413cceb\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " Nov 04 01:33:36.014789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014453 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-podinfo\") pod \"36db7091-781e-4f96-a979-6ad25e0fb53f\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " Nov 04 01:33:36.014789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014513 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36db7091-781e-4f96-a979-6ad25e0fb53f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36db7091-781e-4f96-a979-6ad25e0fb53f-istiod-ca-cert\") pod \"36db7091-781e-4f96-a979-6ad25e0fb53f\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " Nov 04 01:33:36.014789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014545 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-kube-api-access-8g67p\" (UniqueName: \"kubernetes.io/projected/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-kube-api-access-8g67p\") pod \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " Nov 04 01:33:36.014789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014599 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a7dd85-4ad8-4990-8907-cb64e413cceb-kube-api-access-5ntjp\" (UniqueName: \"kubernetes.io/projected/82a7dd85-4ad8-4990-8907-cb64e413cceb-kube-api-access-5ntjp\") pod \"82a7dd85-4ad8-4990-8907-cb64e413cceb\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " Nov 04 01:33:36.014789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014628 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36db7091-781e-4f96-a979-6ad25e0fb53f-kube-api-access-474bf\" (UniqueName: \"kubernetes.io/projected/36db7091-781e-4f96-a979-6ad25e0fb53f-kube-api-access-474bf\") pod \"36db7091-781e-4f96-a979-6ad25e0fb53f\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " Nov 04 01:33:36.014789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014719 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-credential-socket\") pod \"6b03d6b4-7003-4705-b3df-64461213d3bc\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " Nov 04 01:33:36.014789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014769 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b03d6b4-7003-4705-b3df-64461213d3bc-kube-api-access-wglch\" (UniqueName: \"kubernetes.io/projected/6b03d6b4-7003-4705-b3df-64461213d3bc-kube-api-access-wglch\") pod \"6b03d6b4-7003-4705-b3df-64461213d3bc\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " Nov 04 01:33:36.015255 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014814 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-workload-socket\") pod \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " Nov 04 01:33:36.015255 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014884 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-token\" (UniqueName: \"kubernetes.io/projected/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-token\") pod \"36db7091-781e-4f96-a979-6ad25e0fb53f\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " Nov 04 01:33:36.015255 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014943 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-token\" (UniqueName: \"kubernetes.io/projected/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-token\") pod \"6b03d6b4-7003-4705-b3df-64461213d3bc\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " Nov 04 01:33:36.015255 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.014998 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-credential-socket\") pod \"36db7091-781e-4f96-a979-6ad25e0fb53f\" (UID: \"36db7091-781e-4f96-a979-6ad25e0fb53f\") " Nov 04 01:33:36.015255 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.015060 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-workload-socket\") pod \"82a7dd85-4ad8-4990-8907-cb64e413cceb\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " Nov 04 01:33:36.015255 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.015087 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-workload-certs\") pod \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " Nov 04 01:33:36.015670 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.015402 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-data" pod "6b03d6b4-7003-4705-b3df-64461213d3bc" (UID: "6b03d6b4-7003-4705-b3df-64461213d3bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:36.015670 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.015612 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-workload-certs" pod "6b03d6b4-7003-4705-b3df-64461213d3bc" (UID: "6b03d6b4-7003-4705-b3df-64461213d3bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:36.016040 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.015796 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-credential-socket" pod "82a7dd85-4ad8-4990-8907-cb64e413cceb" (UID: "82a7dd85-4ad8-4990-8907-cb64e413cceb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:36.016155 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.015124 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-podinfo\") pod \"6b03d6b4-7003-4705-b3df-64461213d3bc\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " Nov 04 01:33:36.016235 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.016192 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-envoy\") pod \"6b03d6b4-7003-4705-b3df-64461213d3bc\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " Nov 04 01:33:36.016235 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.016230 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82a7dd85-4ad8-4990-8907-cb64e413cceb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a7dd85-4ad8-4990-8907-cb64e413cceb-istiod-ca-cert\") pod \"82a7dd85-4ad8-4990-8907-cb64e413cceb\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " Nov 04 01:33:36.016340 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.016262 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-data\") pod \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " Nov 04 01:33:36.016340 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.016307 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-credential-socket\") pod \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " Nov 04 01:33:36.016451 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.016362 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-workload-certs\") pod \"82a7dd85-4ad8-4990-8907-cb64e413cceb\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " Nov 04 01:33:36.016451 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.016408 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istiod-ca-cert\") pod \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\" (UID: \"a4cd0527-183f-484b-8dc5-8fc84aa45b1b\") " Nov 04 01:33:36.016451 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.016438 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-token\" (UniqueName: \"kubernetes.io/projected/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-token\") pod \"82a7dd85-4ad8-4990-8907-cb64e413cceb\" (UID: \"82a7dd85-4ad8-4990-8907-cb64e413cceb\") " Nov 04 01:33:36.016631 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.016481 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-workload-socket\") pod \"6b03d6b4-7003-4705-b3df-64461213d3bc\" (UID: \"6b03d6b4-7003-4705-b3df-64461213d3bc\") " Nov 04 01:33:36.016943 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.016738 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-workload-socket" pod "36db7091-781e-4f96-a979-6ad25e0fb53f" (UID: "36db7091-781e-4f96-a979-6ad25e0fb53f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:36.019678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.017047 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-workload-socket" pod "6b03d6b4-7003-4705-b3df-64461213d3bc" (UID: "6b03d6b4-7003-4705-b3df-64461213d3bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:36.019678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.018169 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b03d6b4-7003-4705-b3df-64461213d3bc-kube-api-access-wglch" pod "6b03d6b4-7003-4705-b3df-64461213d3bc" (UID: "6b03d6b4-7003-4705-b3df-64461213d3bc"). InnerVolumeSpecName "kube-api-access-wglch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:36.019678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.018462 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b03d6b4-7003-4705-b3df-64461213d3bc-istiod-ca-cert" pod "6b03d6b4-7003-4705-b3df-64461213d3bc" (UID: "6b03d6b4-7003-4705-b3df-64461213d3bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:36.019678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.018654 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-envoy" pod "82a7dd85-4ad8-4990-8907-cb64e413cceb" (UID: "82a7dd85-4ad8-4990-8907-cb64e413cceb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:36.019678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.019192 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-kube-api-access-8g67p" pod "a4cd0527-183f-484b-8dc5-8fc84aa45b1b" (UID: "a4cd0527-183f-484b-8dc5-8fc84aa45b1b"). InnerVolumeSpecName "kube-api-access-8g67p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:36.019989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.019949 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-workload-socket" pod "a4cd0527-183f-484b-8dc5-8fc84aa45b1b" (UID: "a4cd0527-183f-484b-8dc5-8fc84aa45b1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:36.020756 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.020654 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a7dd85-4ad8-4990-8907-cb64e413cceb-kube-api-access-5ntjp" pod "82a7dd85-4ad8-4990-8907-cb64e413cceb" (UID: "82a7dd85-4ad8-4990-8907-cb64e413cceb"). InnerVolumeSpecName "kube-api-access-5ntjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:36.021196 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.020978 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36db7091-781e-4f96-a979-6ad25e0fb53f-istiod-ca-cert" pod "36db7091-781e-4f96-a979-6ad25e0fb53f" (UID: "36db7091-781e-4f96-a979-6ad25e0fb53f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:36.021604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.021543 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a7dd85-4ad8-4990-8907-cb64e413cceb-istiod-ca-cert" pod "82a7dd85-4ad8-4990-8907-cb64e413cceb" (UID: "82a7dd85-4ad8-4990-8907-cb64e413cceb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:36.021772 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.021752 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-podinfo" pod "36db7091-781e-4f96-a979-6ad25e0fb53f" (UID: "36db7091-781e-4f96-a979-6ad25e0fb53f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:36.021968 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.021949 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-workload-certs" pod "82a7dd85-4ad8-4990-8907-cb64e413cceb" (UID: "82a7dd85-4ad8-4990-8907-cb64e413cceb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:36.022092 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.022078 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-data" pod "a4cd0527-183f-484b-8dc5-8fc84aa45b1b" (UID: "a4cd0527-183f-484b-8dc5-8fc84aa45b1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:36.022208 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.022192 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-credential-socket" pod "a4cd0527-183f-484b-8dc5-8fc84aa45b1b" (UID: "a4cd0527-183f-484b-8dc5-8fc84aa45b1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:36.032590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.032176 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-data" pod "36db7091-781e-4f96-a979-6ad25e0fb53f" (UID: "36db7091-781e-4f96-a979-6ad25e0fb53f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:36.032590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.032205 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-credential-socket" pod "6b03d6b4-7003-4705-b3df-64461213d3bc" (UID: "6b03d6b4-7003-4705-b3df-64461213d3bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:36.032590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.032344 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-envoy" pod "a4cd0527-183f-484b-8dc5-8fc84aa45b1b" (UID: "a4cd0527-183f-484b-8dc5-8fc84aa45b1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:36.032590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.032404 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-podinfo" pod "82a7dd85-4ad8-4990-8907-cb64e413cceb" (UID: "82a7dd85-4ad8-4990-8907-cb64e413cceb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:36.032855 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.032652 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-envoy" pod "36db7091-781e-4f96-a979-6ad25e0fb53f" (UID: "36db7091-781e-4f96-a979-6ad25e0fb53f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:36.032855 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.017753 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.032855 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.032768 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.032855 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.032782 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.032855 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.032792 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.032855 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.032801 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.032855 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.032810 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.034917 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.033428 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-credential-socket" pod "36db7091-781e-4f96-a979-6ad25e0fb53f" (UID: "36db7091-781e-4f96-a979-6ad25e0fb53f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:36.034917 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.033443 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-workload-certs" pod "a4cd0527-183f-484b-8dc5-8fc84aa45b1b" (UID: "a4cd0527-183f-484b-8dc5-8fc84aa45b1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:36.034917 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.033902 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istiod-ca-cert" pod "a4cd0527-183f-484b-8dc5-8fc84aa45b1b" (UID: "a4cd0527-183f-484b-8dc5-8fc84aa45b1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:36.034917 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.034417 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-workload-socket" pod "82a7dd85-4ad8-4990-8907-cb64e413cceb" (UID: "82a7dd85-4ad8-4990-8907-cb64e413cceb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:36.037188 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.037079 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-podinfo" pod "6b03d6b4-7003-4705-b3df-64461213d3bc" (UID: "6b03d6b4-7003-4705-b3df-64461213d3bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:36.037338 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.037314 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-envoy" pod "6b03d6b4-7003-4705-b3df-64461213d3bc" (UID: "6b03d6b4-7003-4705-b3df-64461213d3bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:36.040278 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.040211 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-token" pod "6b03d6b4-7003-4705-b3df-64461213d3bc" (UID: "6b03d6b4-7003-4705-b3df-64461213d3bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:36.040627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.040546 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-token" pod "82a7dd85-4ad8-4990-8907-cb64e413cceb" (UID: "82a7dd85-4ad8-4990-8907-cb64e413cceb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:36.040627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.040597 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-podinfo" pod "a4cd0527-183f-484b-8dc5-8fc84aa45b1b" (UID: "a4cd0527-183f-484b-8dc5-8fc84aa45b1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:36.040857 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.040823 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-token" pod "a4cd0527-183f-484b-8dc5-8fc84aa45b1b" (UID: "a4cd0527-183f-484b-8dc5-8fc84aa45b1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:36.040940 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.040869 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-token" pod "36db7091-781e-4f96-a979-6ad25e0fb53f" (UID: "36db7091-781e-4f96-a979-6ad25e0fb53f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:36.041085 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.041066 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36db7091-781e-4f96-a979-6ad25e0fb53f-kube-api-access-474bf" pod "36db7091-781e-4f96-a979-6ad25e0fb53f" (UID: "36db7091-781e-4f96-a979-6ad25e0fb53f"). InnerVolumeSpecName "kube-api-access-474bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:36.133935 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.133906 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.133935 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.133931 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wglch\" (UniqueName: \"kubernetes.io/projected/6b03d6b4-7003-4705-b3df-64461213d3bc-kube-api-access-wglch\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.133941 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.133954 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.133963 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.133972 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.133980 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.133989 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.133998 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.134006 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.134015 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a7dd85-4ad8-4990-8907-cb64e413cceb-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.134024 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.134032 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.134040 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.134047 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.134055 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.134062 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b03d6b4-7003-4705-b3df-64461213d3bc-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.134070 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.134077 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.134084 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.134092 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b03d6b4-7003-4705-b3df-64461213d3bc-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.134101 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.134109 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.134116 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.134123 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a7dd85-4ad8-4990-8907-cb64e413cceb-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.134131 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36db7091-781e-4f96-a979-6ad25e0fb53f-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.134138 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36db7091-781e-4f96-a979-6ad25e0fb53f-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.134147 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g67p\" (UniqueName: \"kubernetes.io/projected/a4cd0527-183f-484b-8dc5-8fc84aa45b1b-kube-api-access-8g67p\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134909 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.134157 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ntjp\" (UniqueName: \"kubernetes.io/projected/82a7dd85-4ad8-4990-8907-cb64e413cceb-kube-api-access-5ntjp\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.134909 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.134165 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-474bf\" (UniqueName: \"kubernetes.io/projected/36db7091-781e-4f96-a979-6ad25e0fb53f-kube-api-access-474bf\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:36.492935 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.492857 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:36.519383 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.519350 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" event={"ID":"82a7dd85-4ad8-4990-8907-cb64e413cceb","Type":"ContainerDied","Data":"1fd54b535ae3cecb6dd3e5c7c86fcdeca7e8748d7f4e1eddf6e31729889889b7"} Nov 04 01:33:36.519610 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.519422 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4" Nov 04 01:33:36.519610 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.519449 4852 scope.go:122] "RemoveContainer" containerID="bd9cbabd2ff2ea49ee861aea08045312dc508f5760614c80ead0293e4dcc7915" Nov 04 01:33:36.527611 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.523105 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" Nov 04 01:33:36.527611 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.523610 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l" event={"ID":"36db7091-781e-4f96-a979-6ad25e0fb53f","Type":"ContainerDied","Data":"b57b55063da83f979951968c4d7e49da118cd6cb13de890da8c35eb837bcd257"} Nov 04 01:33:36.527611 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.525003 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" event={"ID":"a4cd0527-183f-484b-8dc5-8fc84aa45b1b","Type":"ContainerDied","Data":"0966764d92b6e801648f8a66713ecc463f7898d9a339d9720b7aae2806a73422"} Nov 04 01:33:36.527611 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.525108 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m" Nov 04 01:33:36.531963 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.531938 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" event={"ID":"a247fcaf-f6ae-499c-83d9-acfdfa09cad0","Type":"ContainerDied","Data":"324f02e82bcc9e2a6a1214aac718de2e3fd141f0c60e73d70e959c778b7fdedb"} Nov 04 01:33:36.532154 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.532025 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c" Nov 04 01:33:36.536900 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.536884 4852 scope.go:122] "RemoveContainer" containerID="051f67ebf8605d67633e79cdcc778071025ef0d5567155753fdec4b2613434b0" Nov 04 01:33:36.537398 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.537378 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" Nov 04 01:33:36.537703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.537371 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz" event={"ID":"6b03d6b4-7003-4705-b3df-64461213d3bc","Type":"ContainerDied","Data":"2a1eb551c08e9a183f3b3438888f86f3e506efed827cbdfd900cbced43c3ab85"} Nov 04 01:33:36.541645 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.541625 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" event={"ID":"169d147a-6807-4836-8eca-4b7aa3d33bd5","Type":"ContainerDied","Data":"c83179357a361ece76faea6859d63c42fd4b958948cacd01825356758f719e29"} Nov 04 01:33:36.541955 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.541929 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw" Nov 04 01:33:36.551302 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.551281 4852 scope.go:122] "RemoveContainer" containerID="ddd324c2e3ccaa3c0a89a02b2431d59a01c232f50d5e0649a1cf34ac4d2b1964" Nov 04 01:33:36.569614 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.569580 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4"] Nov 04 01:33:36.571320 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.571295 4852 scope.go:122] "RemoveContainer" containerID="7bc34e24d741f58a32e121838c46b49a793759d33225049adffbea76091b4ee2" Nov 04 01:33:36.579411 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.579384 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-858l4"] Nov 04 01:33:36.596011 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.595982 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l"] Nov 04 01:33:36.599805 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.599774 4852 scope.go:122] "RemoveContainer" containerID="7cf793fb503d17ce8969dad595474c0b451a20e88b8c69024d5c2e7264531b9e" Nov 04 01:33:36.600701 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.600677 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4s46l"] Nov 04 01:33:36.611402 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.611385 4852 scope.go:122] "RemoveContainer" containerID="aa8ecffc130835fd35082b1f1ff491a853a2d44bb45b907b64a30b00d6a40562" Nov 04 01:33:36.615225 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.615197 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m"] Nov 04 01:33:36.621535 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.621512 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-fzn6m"] Nov 04 01:33:36.632816 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.632794 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw"] Nov 04 01:33:36.636941 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.636918 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-m6fjw"] Nov 04 01:33:36.654655 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.654632 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c"] Nov 04 01:33:36.656429 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.656407 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-4k62c"] Nov 04 01:33:36.670742 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.670721 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz"] Nov 04 01:33:36.679210 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:36.679189 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-vcvgz"] Nov 04 01:33:37.287092 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:37.286138 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:38.000304 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:38.000270 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="169d147a-6807-4836-8eca-4b7aa3d33bd5" path="/var/lib/kubelet/pods/169d147a-6807-4836-8eca-4b7aa3d33bd5/volumes" Nov 04 01:33:38.002041 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:38.000926 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36db7091-781e-4f96-a979-6ad25e0fb53f" path="/var/lib/kubelet/pods/36db7091-781e-4f96-a979-6ad25e0fb53f/volumes" Nov 04 01:33:38.002041 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:38.001556 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b03d6b4-7003-4705-b3df-64461213d3bc" path="/var/lib/kubelet/pods/6b03d6b4-7003-4705-b3df-64461213d3bc/volumes" Nov 04 01:33:38.002683 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:38.002664 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82a7dd85-4ad8-4990-8907-cb64e413cceb" path="/var/lib/kubelet/pods/82a7dd85-4ad8-4990-8907-cb64e413cceb/volumes" Nov 04 01:33:38.003111 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:38.003098 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a247fcaf-f6ae-499c-83d9-acfdfa09cad0" path="/var/lib/kubelet/pods/a247fcaf-f6ae-499c-83d9-acfdfa09cad0/volumes" Nov 04 01:33:38.003516 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:38.003504 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4cd0527-183f-484b-8dc5-8fc84aa45b1b" path="/var/lib/kubelet/pods/a4cd0527-183f-484b-8dc5-8fc84aa45b1b/volumes" Nov 04 01:33:38.050734 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:38.050703 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:38.202791 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:38.202755 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:38.372760 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:38.372731 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 01:33:38.395681 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:38.395643 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 01:33:38.617353 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:38.610167 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:39.236704 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:39.236674 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:40.029714 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.026614 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:40.083598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.081060 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:33:40.154864 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.154829 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67"] Nov 04 01:33:40.165226 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.165193 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms"] Nov 04 01:33:40.219213 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.219184 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57537b5d-473c-41da-9e67-6c848a66fee7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57537b5d-473c-41da-9e67-6c848a66fee7-istio-podinfo\") pod \"57537b5d-473c-41da-9e67-6c848a66fee7\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " Nov 04 01:33:40.219373 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.219283 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57537b5d-473c-41da-9e67-6c848a66fee7-istio-token\" (UniqueName: \"kubernetes.io/projected/57537b5d-473c-41da-9e67-6c848a66fee7-istio-token\") pod \"57537b5d-473c-41da-9e67-6c848a66fee7\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " Nov 04 01:33:40.219373 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.219306 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-credential-socket\") pod \"57537b5d-473c-41da-9e67-6c848a66fee7\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " Nov 04 01:33:40.219373 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.219335 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57537b5d-473c-41da-9e67-6c848a66fee7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57537b5d-473c-41da-9e67-6c848a66fee7-istiod-ca-cert\") pod \"57537b5d-473c-41da-9e67-6c848a66fee7\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " Nov 04 01:33:40.219373 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.219359 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-workload-socket\") pod \"57537b5d-473c-41da-9e67-6c848a66fee7\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " Nov 04 01:33:40.219520 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.219500 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57537b5d-473c-41da-9e67-6c848a66fee7-kube-api-access-t76wp\" (UniqueName: \"kubernetes.io/projected/57537b5d-473c-41da-9e67-6c848a66fee7-kube-api-access-t76wp\") pod \"57537b5d-473c-41da-9e67-6c848a66fee7\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " Nov 04 01:33:40.219609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.219585 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-istio-envoy\") pod \"57537b5d-473c-41da-9e67-6c848a66fee7\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " Nov 04 01:33:40.219806 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.219618 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-istio-data\") pod \"57537b5d-473c-41da-9e67-6c848a66fee7\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " Nov 04 01:33:40.219806 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.219659 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-workload-certs\") pod \"57537b5d-473c-41da-9e67-6c848a66fee7\" (UID: \"57537b5d-473c-41da-9e67-6c848a66fee7\") " Nov 04 01:33:40.219806 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.219785 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-workload-socket" pod "57537b5d-473c-41da-9e67-6c848a66fee7" (UID: "57537b5d-473c-41da-9e67-6c848a66fee7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:40.219980 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.219858 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57537b5d-473c-41da-9e67-6c848a66fee7-istiod-ca-cert" pod "57537b5d-473c-41da-9e67-6c848a66fee7" (UID: "57537b5d-473c-41da-9e67-6c848a66fee7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:40.220131 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.220011 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-workload-certs" pod "57537b5d-473c-41da-9e67-6c848a66fee7" (UID: "57537b5d-473c-41da-9e67-6c848a66fee7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:40.220131 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.220127 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-istio-data" pod "57537b5d-473c-41da-9e67-6c848a66fee7" (UID: "57537b5d-473c-41da-9e67-6c848a66fee7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:40.220303 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.220225 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-credential-socket" pod "57537b5d-473c-41da-9e67-6c848a66fee7" (UID: "57537b5d-473c-41da-9e67-6c848a66fee7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:40.220630 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.220541 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:40.220630 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.220581 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57537b5d-473c-41da-9e67-6c848a66fee7-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:40.220630 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.220594 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:40.220630 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.220603 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:40.220630 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.220612 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:40.221651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.221631 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57537b5d-473c-41da-9e67-6c848a66fee7-istio-token" pod "57537b5d-473c-41da-9e67-6c848a66fee7" (UID: "57537b5d-473c-41da-9e67-6c848a66fee7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:40.222015 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.221993 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57537b5d-473c-41da-9e67-6c848a66fee7-istio-podinfo" pod "57537b5d-473c-41da-9e67-6c848a66fee7" (UID: "57537b5d-473c-41da-9e67-6c848a66fee7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:40.222225 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.222208 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57537b5d-473c-41da-9e67-6c848a66fee7-kube-api-access-t76wp" pod "57537b5d-473c-41da-9e67-6c848a66fee7" (UID: "57537b5d-473c-41da-9e67-6c848a66fee7"). InnerVolumeSpecName "kube-api-access-t76wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:40.222509 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.222492 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-istio-envoy" pod "57537b5d-473c-41da-9e67-6c848a66fee7" (UID: "57537b5d-473c-41da-9e67-6c848a66fee7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:40.298869 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.298822 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:40.321797 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.321769 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57537b5d-473c-41da-9e67-6c848a66fee7-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:40.321797 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.321794 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57537b5d-473c-41da-9e67-6c848a66fee7-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:40.321987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.321805 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57537b5d-473c-41da-9e67-6c848a66fee7-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:40.321987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.321814 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t76wp\" (UniqueName: \"kubernetes.io/projected/57537b5d-473c-41da-9e67-6c848a66fee7-kube-api-access-t76wp\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:40.593076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.592970 4852 generic.go:356] "Generic (PLEG): container finished" podID="57537b5d-473c-41da-9e67-6c848a66fee7" containerID="1d6201d9c2ff6e099ef6ac5252c7c828c889c83a42af01358c2e567bb839831e" exitCode=0 Nov 04 01:33:40.593076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.593015 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" Nov 04 01:33:40.593076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.593061 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" event={"ID":"57537b5d-473c-41da-9e67-6c848a66fee7","Type":"ContainerDied","Data":"1d6201d9c2ff6e099ef6ac5252c7c828c889c83a42af01358c2e567bb839831e"} Nov 04 01:33:40.593381 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.593128 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb" event={"ID":"57537b5d-473c-41da-9e67-6c848a66fee7","Type":"ContainerDied","Data":"385cc8d26d87915838ec18b101443ffdd81d28139ca3881531c8dedf36974e9c"} Nov 04 01:33:40.593381 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.593158 4852 scope.go:122] "RemoveContainer" containerID="1d6201d9c2ff6e099ef6ac5252c7c828c889c83a42af01358c2e567bb839831e" Nov 04 01:33:40.604605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.604587 4852 scope.go:122] "RemoveContainer" containerID="0205cb769edf06deebde37199e8ff5561c058fc76dbd456b0b36b62c04e5360e" Nov 04 01:33:40.621796 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.618527 4852 scope.go:122] "RemoveContainer" containerID="1d6201d9c2ff6e099ef6ac5252c7c828c889c83a42af01358c2e567bb839831e" Nov 04 01:33:40.628819 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.626809 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb"] Nov 04 01:33:40.629179 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:33:40.628410 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d6201d9c2ff6e099ef6ac5252c7c828c889c83a42af01358c2e567bb839831e\": container with ID starting with 1d6201d9c2ff6e099ef6ac5252c7c828c889c83a42af01358c2e567bb839831e not found: ID does not exist" containerID="1d6201d9c2ff6e099ef6ac5252c7c828c889c83a42af01358c2e567bb839831e" Nov 04 01:33:40.629179 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.628876 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d6201d9c2ff6e099ef6ac5252c7c828c889c83a42af01358c2e567bb839831e"} err="failed to get container status \"1d6201d9c2ff6e099ef6ac5252c7c828c889c83a42af01358c2e567bb839831e\": rpc error: code = NotFound desc = could not find container \"1d6201d9c2ff6e099ef6ac5252c7c828c889c83a42af01358c2e567bb839831e\": container with ID starting with 1d6201d9c2ff6e099ef6ac5252c7c828c889c83a42af01358c2e567bb839831e not found: ID does not exist" Nov 04 01:33:40.629179 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.628906 4852 scope.go:122] "RemoveContainer" containerID="0205cb769edf06deebde37199e8ff5561c058fc76dbd456b0b36b62c04e5360e" Nov 04 01:33:40.633970 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:33:40.633936 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0205cb769edf06deebde37199e8ff5561c058fc76dbd456b0b36b62c04e5360e\": container with ID starting with 0205cb769edf06deebde37199e8ff5561c058fc76dbd456b0b36b62c04e5360e not found: ID does not exist" containerID="0205cb769edf06deebde37199e8ff5561c058fc76dbd456b0b36b62c04e5360e" Nov 04 01:33:40.634095 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.633978 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0205cb769edf06deebde37199e8ff5561c058fc76dbd456b0b36b62c04e5360e"} err="failed to get container status \"0205cb769edf06deebde37199e8ff5561c058fc76dbd456b0b36b62c04e5360e\": rpc error: code = NotFound desc = could not find container \"0205cb769edf06deebde37199e8ff5561c058fc76dbd456b0b36b62c04e5360e\": container with ID starting with 0205cb769edf06deebde37199e8ff5561c058fc76dbd456b0b36b62c04e5360e not found: ID does not exist" Nov 04 01:33:40.635005 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:40.634984 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5f572a-openshift-default-5c7b9ddc7c-hntbb"] Nov 04 01:33:41.055209 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:41.055165 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:41.544756 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:41.544721 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts"] Nov 04 01:33:41.556033 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:41.556006 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc"] Nov 04 01:33:41.606039 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:41.605957 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:41.608580 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:41.608543 4852 generic.go:356] "Generic (PLEG): container finished" podID="6604612f-7353-40ab-8a1b-29b19b3395f0" containerID="61ba8bb1d7fb264cb90f65de8d21537ccc1d63ee031575b78765002f4e599288" exitCode=0 Nov 04 01:33:41.608730 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:41.608681 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" event={"ID":"6604612f-7353-40ab-8a1b-29b19b3395f0","Type":"ContainerDied","Data":"61ba8bb1d7fb264cb90f65de8d21537ccc1d63ee031575b78765002f4e599288"} Nov 04 01:33:41.609093 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:41.609074 4852 scope.go:122] "RemoveContainer" containerID="61ba8bb1d7fb264cb90f65de8d21537ccc1d63ee031575b78765002f4e599288" Nov 04 01:33:41.611333 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:41.611291 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:33:41.611709 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:41.611381 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:33:41.611709 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:41.611430 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:33:42.011596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:42.010058 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57537b5d-473c-41da-9e67-6c848a66fee7" path="/var/lib/kubelet/pods/57537b5d-473c-41da-9e67-6c848a66fee7/volumes" Nov 04 01:33:42.617185 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:42.617155 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" event={"ID":"6604612f-7353-40ab-8a1b-29b19b3395f0","Type":"ContainerStarted","Data":"2683c5f12c7114cd0d61b5b8aafe42e5cc34403fcf1618a30ce756202182f4ff"} Nov 04 01:33:42.907401 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:42.907305 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h"] Nov 04 01:33:42.907816 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:42.907778 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" podUID="cf3ef542-79a2-485f-b8ae-0abd09b05734" containerName="maas-api" containerID="cri-o://3a42f37bf986f629b2b88b5b90555913d4ab8aba590b0c9d0ecc240d24a7bfc8" gracePeriod=30 Nov 04 01:33:43.348778 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.341942 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" Nov 04 01:33:43.410385 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.406711 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf3ef542-79a2-485f-b8ae-0abd09b05734-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf3ef542-79a2-485f-b8ae-0abd09b05734-maas-api-tls\") pod \"cf3ef542-79a2-485f-b8ae-0abd09b05734\" (UID: \"cf3ef542-79a2-485f-b8ae-0abd09b05734\") " Nov 04 01:33:43.423620 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.420037 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf3ef542-79a2-485f-b8ae-0abd09b05734-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf3ef542-79a2-485f-b8ae-0abd09b05734-metrics-certs\") pod \"cf3ef542-79a2-485f-b8ae-0abd09b05734\" (UID: \"cf3ef542-79a2-485f-b8ae-0abd09b05734\") " Nov 04 01:33:43.423620 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.420168 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf3ef542-79a2-485f-b8ae-0abd09b05734-kube-api-access-t8xtg\" (UniqueName: \"kubernetes.io/projected/cf3ef542-79a2-485f-b8ae-0abd09b05734-kube-api-access-t8xtg\") pod \"cf3ef542-79a2-485f-b8ae-0abd09b05734\" (UID: \"cf3ef542-79a2-485f-b8ae-0abd09b05734\") " Nov 04 01:33:43.423620 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.420210 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf3ef542-79a2-485f-b8ae-0abd09b05734-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf3ef542-79a2-485f-b8ae-0abd09b05734-openshift-service-ca\") pod \"cf3ef542-79a2-485f-b8ae-0abd09b05734\" (UID: \"cf3ef542-79a2-485f-b8ae-0abd09b05734\") " Nov 04 01:33:43.434599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.430787 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf3ef542-79a2-485f-b8ae-0abd09b05734-openshift-service-ca" pod "cf3ef542-79a2-485f-b8ae-0abd09b05734" (UID: "cf3ef542-79a2-485f-b8ae-0abd09b05734"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:43.522600 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.520669 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:33:43.529587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.525852 4852 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf3ef542-79a2-485f-b8ae-0abd09b05734-openshift-service-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:43.546692 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.538785 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3ef542-79a2-485f-b8ae-0abd09b05734-kube-api-access-t8xtg" pod "cf3ef542-79a2-485f-b8ae-0abd09b05734" (UID: "cf3ef542-79a2-485f-b8ae-0abd09b05734"). InnerVolumeSpecName "kube-api-access-t8xtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:43.546692 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.545322 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf3ef542-79a2-485f-b8ae-0abd09b05734-maas-api-tls" pod "cf3ef542-79a2-485f-b8ae-0abd09b05734" (UID: "cf3ef542-79a2-485f-b8ae-0abd09b05734"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:43.554971 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.552695 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf3ef542-79a2-485f-b8ae-0abd09b05734-metrics-certs" pod "cf3ef542-79a2-485f-b8ae-0abd09b05734" (UID: "cf3ef542-79a2-485f-b8ae-0abd09b05734"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:43.584646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.580109 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:33:43.584646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.581635 4852 patch_prober.go:29] interesting pod/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 04 01:33:43.584646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.581706 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" podUID="6604612f-7353-40ab-8a1b-29b19b3395f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 04 01:33:43.624899 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.624867 4852 generic.go:356] "Generic (PLEG): container finished" podID="cf3ef542-79a2-485f-b8ae-0abd09b05734" containerID="3a42f37bf986f629b2b88b5b90555913d4ab8aba590b0c9d0ecc240d24a7bfc8" exitCode=0 Nov 04 01:33:43.626348 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.624925 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" Nov 04 01:33:43.626348 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.624982 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" event={"ID":"cf3ef542-79a2-485f-b8ae-0abd09b05734","Type":"ContainerDied","Data":"3a42f37bf986f629b2b88b5b90555913d4ab8aba590b0c9d0ecc240d24a7bfc8"} Nov 04 01:33:43.626348 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.625023 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h" event={"ID":"cf3ef542-79a2-485f-b8ae-0abd09b05734","Type":"ContainerDied","Data":"6f1a9aa277768bbd17d4c744e6f90ee9b84a8ca48b21bdd3cd65efc411cea5e0"} Nov 04 01:33:43.626348 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.625049 4852 scope.go:122] "RemoveContainer" containerID="3a42f37bf986f629b2b88b5b90555913d4ab8aba590b0c9d0ecc240d24a7bfc8" Nov 04 01:33:43.627206 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.627052 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf3ef542-79a2-485f-b8ae-0abd09b05734-maas-api-tls\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:43.627206 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.627076 4852 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf3ef542-79a2-485f-b8ae-0abd09b05734-metrics-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:43.627206 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.627090 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8xtg\" (UniqueName: \"kubernetes.io/projected/cf3ef542-79a2-485f-b8ae-0abd09b05734-kube-api-access-t8xtg\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:43.637823 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.637804 4852 scope.go:122] "RemoveContainer" containerID="3a42f37bf986f629b2b88b5b90555913d4ab8aba590b0c9d0ecc240d24a7bfc8" Nov 04 01:33:43.638115 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:33:43.638098 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a42f37bf986f629b2b88b5b90555913d4ab8aba590b0c9d0ecc240d24a7bfc8\": container with ID starting with 3a42f37bf986f629b2b88b5b90555913d4ab8aba590b0c9d0ecc240d24a7bfc8 not found: ID does not exist" containerID="3a42f37bf986f629b2b88b5b90555913d4ab8aba590b0c9d0ecc240d24a7bfc8" Nov 04 01:33:43.638160 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.638124 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a42f37bf986f629b2b88b5b90555913d4ab8aba590b0c9d0ecc240d24a7bfc8"} err="failed to get container status \"3a42f37bf986f629b2b88b5b90555913d4ab8aba590b0c9d0ecc240d24a7bfc8\": rpc error: code = NotFound desc = could not find container \"3a42f37bf986f629b2b88b5b90555913d4ab8aba590b0c9d0ecc240d24a7bfc8\": container with ID starting with 3a42f37bf986f629b2b88b5b90555913d4ab8aba590b0c9d0ecc240d24a7bfc8 not found: ID does not exist" Nov 04 01:33:43.655879 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.655860 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h"] Nov 04 01:33:43.662614 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:43.660275 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c75ea1-58c8d5f97d-zn28h"] Nov 04 01:33:44.056729 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.054666 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf3ef542-79a2-485f-b8ae-0abd09b05734" path="/var/lib/kubelet/pods/cf3ef542-79a2-485f-b8ae-0abd09b05734/volumes" Nov 04 01:33:44.412946 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.412914 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9"] Nov 04 01:33:44.413641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413621 4852 cpu_manager.go:414] "containerMap: removing container" podUID="57537b5d-473c-41da-9e67-6c848a66fee7" containerName="istio-proxy" Nov 04 01:33:44.413775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413642 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57537b5d-473c-41da-9e67-6c848a66fee7" containerName="istio-proxy" Nov 04 01:33:44.413775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413661 4852 cpu_manager.go:414] "containerMap: removing container" podUID="784fa5a1-0f23-4a6e-a6c4-eae42e88f7da" containerName="istio-proxy" Nov 04 01:33:44.413775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413668 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784fa5a1-0f23-4a6e-a6c4-eae42e88f7da" containerName="istio-proxy" Nov 04 01:33:44.413775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413696 4852 cpu_manager.go:414] "containerMap: removing container" podUID="6b03d6b4-7003-4705-b3df-64461213d3bc" containerName="istio-proxy" Nov 04 01:33:44.413775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413703 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b03d6b4-7003-4705-b3df-64461213d3bc" containerName="istio-proxy" Nov 04 01:33:44.413775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413717 4852 cpu_manager.go:414] "containerMap: removing container" podUID="82a7dd85-4ad8-4990-8907-cb64e413cceb" containerName="istio-proxy" Nov 04 01:33:44.413775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413724 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82a7dd85-4ad8-4990-8907-cb64e413cceb" containerName="istio-proxy" Nov 04 01:33:44.413775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413738 4852 cpu_manager.go:414] "containerMap: removing container" podUID="a247fcaf-f6ae-499c-83d9-acfdfa09cad0" containerName="istio-proxy" Nov 04 01:33:44.413775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413744 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a247fcaf-f6ae-499c-83d9-acfdfa09cad0" containerName="istio-proxy" Nov 04 01:33:44.413775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413765 4852 cpu_manager.go:414] "containerMap: removing container" podUID="a4cd0527-183f-484b-8dc5-8fc84aa45b1b" containerName="istio-proxy" Nov 04 01:33:44.413775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413771 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4cd0527-183f-484b-8dc5-8fc84aa45b1b" containerName="istio-proxy" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413789 4852 cpu_manager.go:414] "containerMap: removing container" podUID="2af6084e-6b1b-4728-b4d0-6ca48db7a30b" containerName="istio-proxy" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413795 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2af6084e-6b1b-4728-b4d0-6ca48db7a30b" containerName="istio-proxy" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413806 4852 cpu_manager.go:414] "containerMap: removing container" podUID="f244db1e-5289-4885-9e1c-2b63731821ca" containerName="istio-proxy" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413813 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f244db1e-5289-4885-9e1c-2b63731821ca" containerName="istio-proxy" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413837 4852 cpu_manager.go:414] "containerMap: removing container" podUID="cf3ef542-79a2-485f-b8ae-0abd09b05734" containerName="maas-api" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413844 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf3ef542-79a2-485f-b8ae-0abd09b05734" containerName="maas-api" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413864 4852 cpu_manager.go:414] "containerMap: removing container" podUID="36db7091-781e-4f96-a979-6ad25e0fb53f" containerName="istio-proxy" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413871 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36db7091-781e-4f96-a979-6ad25e0fb53f" containerName="istio-proxy" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413882 4852 cpu_manager.go:414] "containerMap: removing container" podUID="93308cb8-8e7f-45c3-a1f6-bbd045723e67" containerName="istio-proxy" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413889 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93308cb8-8e7f-45c3-a1f6-bbd045723e67" containerName="istio-proxy" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413906 4852 cpu_manager.go:414] "containerMap: removing container" podUID="169d147a-6807-4836-8eca-4b7aa3d33bd5" containerName="istio-proxy" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.413912 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169d147a-6807-4836-8eca-4b7aa3d33bd5" containerName="istio-proxy" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.414051 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf3ef542-79a2-485f-b8ae-0abd09b05734" containerName="maas-api" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.414067 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="36db7091-781e-4f96-a979-6ad25e0fb53f" containerName="istio-proxy" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.414075 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="93308cb8-8e7f-45c3-a1f6-bbd045723e67" containerName="istio-proxy" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.414086 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="57537b5d-473c-41da-9e67-6c848a66fee7" containerName="istio-proxy" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.414097 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="169d147a-6807-4836-8eca-4b7aa3d33bd5" containerName="istio-proxy" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.414105 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="57537b5d-473c-41da-9e67-6c848a66fee7" containerName="istio-proxy" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.414114 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="784fa5a1-0f23-4a6e-a6c4-eae42e88f7da" containerName="istio-proxy" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.414125 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b03d6b4-7003-4705-b3df-64461213d3bc" containerName="istio-proxy" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.414134 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="82a7dd85-4ad8-4990-8907-cb64e413cceb" containerName="istio-proxy" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.414143 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="a247fcaf-f6ae-499c-83d9-acfdfa09cad0" containerName="istio-proxy" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.414153 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4cd0527-183f-484b-8dc5-8fc84aa45b1b" containerName="istio-proxy" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.414164 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="2af6084e-6b1b-4728-b4d0-6ca48db7a30b" containerName="istio-proxy" Nov 04 01:33:44.414325 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.414172 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="f244db1e-5289-4885-9e1c-2b63731821ca" containerName="istio-proxy" Nov 04 01:33:44.419443 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.419420 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.428722 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.428137 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c75ea1-openshift-default-dockercfg-7g542\"" Nov 04 01:33:44.429448 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.429412 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9"] Nov 04 01:33:44.456158 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.456126 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-workload-certs\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.456296 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.456169 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-envoy\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.456296 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.456238 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twmpp\" (UniqueName: \"kubernetes.io/projected/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-kube-api-access-twmpp\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.456454 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.456342 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-workload-socket\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.456531 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.456510 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-podinfo\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.456672 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.456595 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-token\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.456747 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.456667 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-credential-socket\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.456747 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.456698 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-data\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.456854 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.456753 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istiod-ca-cert\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.581958 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.581918 4852 patch_prober.go:29] interesting pod/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 04 01:33:44.582142 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.581986 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" podUID="6604612f-7353-40ab-8a1b-29b19b3395f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 04 01:33:44.583218 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.583190 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-credential-socket\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.583342 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.583228 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-data\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.583342 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.583283 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istiod-ca-cert\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.583471 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.583354 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-workload-certs\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.583471 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.583377 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-envoy\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.583471 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.583412 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twmpp\" (UniqueName: \"kubernetes.io/projected/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-kube-api-access-twmpp\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.583471 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.583446 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-workload-socket\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.583732 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.583675 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-podinfo\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.583732 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.583696 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-credential-socket\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.583840 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.583730 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-token\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.584131 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.584106 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istiod-ca-cert\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.584223 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.584171 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-workload-socket\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.584280 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.584227 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-workload-certs\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.584672 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.584653 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-data\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.586515 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.586496 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-envoy\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.587018 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.587000 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-podinfo\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.593167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.593139 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-token\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.593301 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.593286 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twmpp\" (UniqueName: \"kubernetes.io/projected/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-kube-api-access-twmpp\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:44.684071 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.683962 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:44.746601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:44.735874 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:45.186489 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:45.186415 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9"] Nov 04 01:33:45.186829 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:33:45.186800 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852ff9e8_47a9_4144_b0a1_ff5f9c9cb6e0.slice/crio-f88637aec3e9c996b416e81645165a0aa9398e6a2292862c46163a1117a13bf3 WatchSource:0}: Error finding container f88637aec3e9c996b416e81645165a0aa9398e6a2292862c46163a1117a13bf3: Status 404 returned error can't find the container with id f88637aec3e9c996b416e81645165a0aa9398e6a2292862c46163a1117a13bf3 Nov 04 01:33:45.195838 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:45.195796 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:33:45.195955 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:45.195874 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:33:45.195955 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:45.195926 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:33:45.583336 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:45.582896 4852 patch_prober.go:29] interesting pod/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 04 01:33:45.583336 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:45.582956 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" podUID="6604612f-7353-40ab-8a1b-29b19b3395f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 04 01:33:45.723279 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:45.704758 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:45.751116 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:45.751082 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" event={"ID":"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0","Type":"ContainerStarted","Data":"db0fa2dbac755bdb6c9d80c39f7beff25a73ef974a24bffac39465f2db5f9391"} Nov 04 01:33:45.751284 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:45.751163 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" event={"ID":"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0","Type":"ContainerStarted","Data":"f88637aec3e9c996b416e81645165a0aa9398e6a2292862c46163a1117a13bf3"} Nov 04 01:33:46.016369 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:46.016337 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:46.451413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:46.451294 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:46.546042 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:46.546001 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts" podUID="2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4" containerName="payload-processing" containerID="cri-o://8dc8cf3b3f7d5550644d81a87c0e04f205f3ee6fcce5c2a6ff116955c893ae90" gracePeriod=25 Nov 04 01:33:46.562065 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:46.560762 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc" podUID="68127737-8431-4191-ad9a-d16e080a2ec5" containerName="payload-pre-processing" containerID="cri-o://fab2c9d069534d4e7bc70e93ae3b1ef293c1bf1f11eb958a6f40eb141fd19dc5" gracePeriod=25 Nov 04 01:33:46.581157 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:46.580693 4852 patch_prober.go:29] interesting pod/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 04 01:33:46.581157 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:46.580753 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" podUID="6604612f-7353-40ab-8a1b-29b19b3395f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 04 01:33:46.740638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:46.740542 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:46.742893 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:46.742626 4852 patch_prober.go:29] interesting pod/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:33:46.742893 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:46.742680 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" podUID="852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 04 01:33:46.819079 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:46.819034 4852 generic.go:356] "Generic (PLEG): container finished" podID="68127737-8431-4191-ad9a-d16e080a2ec5" containerID="fab2c9d069534d4e7bc70e93ae3b1ef293c1bf1f11eb958a6f40eb141fd19dc5" exitCode=0 Nov 04 01:33:46.819510 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:46.819474 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc" event={"ID":"68127737-8431-4191-ad9a-d16e080a2ec5","Type":"ContainerDied","Data":"fab2c9d069534d4e7bc70e93ae3b1ef293c1bf1f11eb958a6f40eb141fd19dc5"} Nov 04 01:33:46.821383 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:46.821365 4852 generic.go:356] "Generic (PLEG): container finished" podID="2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4" containerID="8dc8cf3b3f7d5550644d81a87c0e04f205f3ee6fcce5c2a6ff116955c893ae90" exitCode=0 Nov 04 01:33:46.821698 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:46.821682 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts" event={"ID":"2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4","Type":"ContainerDied","Data":"8dc8cf3b3f7d5550644d81a87c0e04f205f3ee6fcce5c2a6ff116955c893ae90"} Nov 04 01:33:47.073597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.065770 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc" Nov 04 01:33:47.109252 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.105889 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" podStartSLOduration=3.105870814 podStartE2EDuration="3.105870814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:33:45.788894854 +0000 UTC m=+2107.904004973" watchObservedRunningTime="2026-11-04 01:33:47.105870814 +0000 UTC m=+2109.220980934" Nov 04 01:33:47.135011 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.134985 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts" Nov 04 01:33:47.215354 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.215304 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4-plugins-config-volume\") pod \"2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4\" (UID: \"2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4\") " Nov 04 01:33:47.215794 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.215770 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68127737-8431-4191-ad9a-d16e080a2ec5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68127737-8431-4191-ad9a-d16e080a2ec5-plugins-config-volume\") pod \"68127737-8431-4191-ad9a-d16e080a2ec5\" (UID: \"68127737-8431-4191-ad9a-d16e080a2ec5\") " Nov 04 01:33:47.216028 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.216012 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68127737-8431-4191-ad9a-d16e080a2ec5-kube-api-access-5wt2m\" (UniqueName: \"kubernetes.io/projected/68127737-8431-4191-ad9a-d16e080a2ec5-kube-api-access-5wt2m\") pod \"68127737-8431-4191-ad9a-d16e080a2ec5\" (UID: \"68127737-8431-4191-ad9a-d16e080a2ec5\") " Nov 04 01:33:47.216182 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.216165 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4-kube-api-access-cl4j6\" (UniqueName: \"kubernetes.io/projected/2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4-kube-api-access-cl4j6\") pod \"2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4\" (UID: \"2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4\") " Nov 04 01:33:47.218098 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.218063 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4-plugins-config-volume" pod "2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4" (UID: "2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:47.218635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.218607 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68127737-8431-4191-ad9a-d16e080a2ec5-plugins-config-volume" pod "68127737-8431-4191-ad9a-d16e080a2ec5" (UID: "68127737-8431-4191-ad9a-d16e080a2ec5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:47.220213 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.220192 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4-kube-api-access-cl4j6" pod "2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4" (UID: "2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4"). InnerVolumeSpecName "kube-api-access-cl4j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:47.221582 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.221550 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68127737-8431-4191-ad9a-d16e080a2ec5-kube-api-access-5wt2m" pod "68127737-8431-4191-ad9a-d16e080a2ec5" (UID: "68127737-8431-4191-ad9a-d16e080a2ec5"). InnerVolumeSpecName "kube-api-access-5wt2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:47.319047 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.318961 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68127737-8431-4191-ad9a-d16e080a2ec5-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:47.319047 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.318992 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wt2m\" (UniqueName: \"kubernetes.io/projected/68127737-8431-4191-ad9a-d16e080a2ec5-kube-api-access-5wt2m\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:47.319047 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.319003 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl4j6\" (UniqueName: \"kubernetes.io/projected/2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4-kube-api-access-cl4j6\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:47.319047 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.319012 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:47.385331 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.385291 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:47.587551 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.587463 4852 patch_prober.go:29] interesting pod/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 04 01:33:47.587822 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.587796 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" podUID="6604612f-7353-40ab-8a1b-29b19b3395f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 04 01:33:47.741283 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.741247 4852 patch_prober.go:29] interesting pod/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:33:47.741726 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.741298 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" podUID="852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 04 01:33:47.829526 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.829502 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc" Nov 04 01:33:47.829755 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.829529 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc" event={"ID":"68127737-8431-4191-ad9a-d16e080a2ec5","Type":"ContainerDied","Data":"ca004a5c30c7092386e16d63f09b8a9f2a94637cf11cb3420c79cb42f48be29b"} Nov 04 01:33:47.829755 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.829597 4852 scope.go:122] "RemoveContainer" containerID="fab2c9d069534d4e7bc70e93ae3b1ef293c1bf1f11eb958a6f40eb141fd19dc5" Nov 04 01:33:47.831684 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.831668 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts" Nov 04 01:33:47.831973 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.831952 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts" event={"ID":"2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4","Type":"ContainerDied","Data":"de54f8544006673bf05119b177312589c3cb973717981ac5561e498e31fe2962"} Nov 04 01:33:47.842275 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.842257 4852 scope.go:122] "RemoveContainer" containerID="8dc8cf3b3f7d5550644d81a87c0e04f205f3ee6fcce5c2a6ff116955c893ae90" Nov 04 01:33:47.907705 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.907653 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc"] Nov 04 01:33:47.910170 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.910149 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bfc54d-5c5cc798f9-t4qnc"] Nov 04 01:33:47.913413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.913398 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts"] Nov 04 01:33:47.915960 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:47.915940 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bfc54d-85bd8779bd-db4ts"] Nov 04 01:33:48.014283 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:48.014248 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4" path="/var/lib/kubelet/pods/2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4/volumes" Nov 04 01:33:48.014776 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:48.014762 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68127737-8431-4191-ad9a-d16e080a2ec5" path="/var/lib/kubelet/pods/68127737-8431-4191-ad9a-d16e080a2ec5/volumes" Nov 04 01:33:48.309592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:48.305687 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:48.585033 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:48.584942 4852 patch_prober.go:29] interesting pod/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 04 01:33:48.585294 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:48.585262 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" podUID="6604612f-7353-40ab-8a1b-29b19b3395f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 04 01:33:48.741232 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:48.741194 4852 patch_prober.go:29] interesting pod/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:33:48.741475 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:48.741453 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" podUID="852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 04 01:33:49.140435 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:49.140400 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:49.344589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:49.336288 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:49.586241 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:49.586204 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:33:49.587684 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:49.587663 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:33:49.740965 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:49.740928 4852 patch_prober.go:29] interesting pod/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:33:49.741123 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:49.740979 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" podUID="852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 04 01:33:49.876638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:49.874225 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:50.096702 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:50.096652 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:50.153657 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:50.145758 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9"] Nov 04 01:33:50.153657 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:50.145982 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" podUID="852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0" containerName="istio-proxy" containerID="cri-o://db0fa2dbac755bdb6c9d80c39f7beff25a73ef974a24bffac39465f2db5f9391" gracePeriod=30 Nov 04 01:33:50.357588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:50.356801 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:50.703905 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:50.703873 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:51.136079 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:51.136045 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:51.321219 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:51.321179 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:51.572110 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:51.572075 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:51.813590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:51.813014 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:52.085236 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:52.085204 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:52.279599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:52.278644 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:52.697615 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:52.694898 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:53.328050 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:53.328012 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:53.510443 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:53.510404 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:54.025674 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:54.025445 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:54.697858 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:54.697822 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:54.883446 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:54.883414 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:55.185141 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.185107 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" containerID="cri-o://781b20931150a22ce2e7ce31ae0710f45917118500c93e1e9c319c5b3881a48f" gracePeriod=45 Nov 04 01:33:55.197994 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.197966 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" containerID="cri-o://a5e4847f710c596a6473fff78d95944d7d4a81e35b0ea084d344b6e6830ff388" gracePeriod=45 Nov 04 01:33:55.433825 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.433741 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": dial tcp 10.134.0.31:8000: connect: connection refused" Nov 04 01:33:55.457997 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.457627 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:55.574879 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.574546 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:33:55.603900 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.603868 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:33:55.624877 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.624830 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-tmp-dir\") pod \"c69e53e0-661a-4a14-94de-d4450a554c80\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " Nov 04 01:33:55.625685 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.625106 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-tmp-dir" pod "c69e53e0-661a-4a14-94de-d4450a554c80" (UID: "c69e53e0-661a-4a14-94de-d4450a554c80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:55.625685 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.625120 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c69e53e0-661a-4a14-94de-d4450a554c80-kube-api-access-pgpfr\" (UniqueName: \"kubernetes.io/projected/c69e53e0-661a-4a14-94de-d4450a554c80-kube-api-access-pgpfr\") pod \"c69e53e0-661a-4a14-94de-d4450a554c80\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " Nov 04 01:33:55.626088 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.626056 4852 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-tmp-dir\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:55.630303 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.630175 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c69e53e0-661a-4a14-94de-d4450a554c80-kube-api-access-pgpfr" pod "c69e53e0-661a-4a14-94de-d4450a554c80" (UID: "c69e53e0-661a-4a14-94de-d4450a554c80"). InnerVolumeSpecName "kube-api-access-pgpfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:55.635577 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.635536 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6"] Nov 04 01:33:55.727622 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.727521 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-home\" (UniqueName: \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-home\") pod \"9211f266-8d05-4bd8-9947-7893a9c0634b\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " Nov 04 01:33:55.727622 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.727597 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-tmp-dir\") pod \"9211f266-8d05-4bd8-9947-7893a9c0634b\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " Nov 04 01:33:55.728133 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.727643 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-dshm\") pod \"c69e53e0-661a-4a14-94de-d4450a554c80\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " Nov 04 01:33:55.728133 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.727898 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9211f266-8d05-4bd8-9947-7893a9c0634b-kube-api-access-mv2cs\" (UniqueName: \"kubernetes.io/projected/9211f266-8d05-4bd8-9947-7893a9c0634b-kube-api-access-mv2cs\") pod \"9211f266-8d05-4bd8-9947-7893a9c0634b\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " Nov 04 01:33:55.728133 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.727956 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-home" pod "9211f266-8d05-4bd8-9947-7893a9c0634b" (UID: "9211f266-8d05-4bd8-9947-7893a9c0634b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:55.728133 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.727972 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-tmp-dir" pod "9211f266-8d05-4bd8-9947-7893a9c0634b" (UID: "9211f266-8d05-4bd8-9947-7893a9c0634b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:55.728133 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.727985 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c69e53e0-661a-4a14-94de-d4450a554c80-tls-certs\" (UniqueName: \"kubernetes.io/secret/c69e53e0-661a-4a14-94de-d4450a554c80-tls-certs\") pod \"c69e53e0-661a-4a14-94de-d4450a554c80\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " Nov 04 01:33:55.728133 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.728022 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-home\" (UniqueName: \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-home\") pod \"c69e53e0-661a-4a14-94de-d4450a554c80\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " Nov 04 01:33:55.728133 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.728087 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-model-cache\") pod \"9211f266-8d05-4bd8-9947-7893a9c0634b\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " Nov 04 01:33:55.728133 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.728118 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-model-cache\") pod \"c69e53e0-661a-4a14-94de-d4450a554c80\" (UID: \"c69e53e0-661a-4a14-94de-d4450a554c80\") " Nov 04 01:33:55.728548 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.728143 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9211f266-8d05-4bd8-9947-7893a9c0634b-tls-certs\" (UniqueName: \"kubernetes.io/secret/9211f266-8d05-4bd8-9947-7893a9c0634b-tls-certs\") pod \"9211f266-8d05-4bd8-9947-7893a9c0634b\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " Nov 04 01:33:55.728548 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.728169 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-dshm\") pod \"9211f266-8d05-4bd8-9947-7893a9c0634b\" (UID: \"9211f266-8d05-4bd8-9947-7893a9c0634b\") " Nov 04 01:33:55.728548 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.728385 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-home" pod "c69e53e0-661a-4a14-94de-d4450a554c80" (UID: "c69e53e0-661a-4a14-94de-d4450a554c80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:55.728833 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.728516 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-model-cache" pod "c69e53e0-661a-4a14-94de-d4450a554c80" (UID: "c69e53e0-661a-4a14-94de-d4450a554c80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:55.729419 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.729393 4852 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-home\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:55.729419 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.729422 4852 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-model-cache\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:55.729419 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.729440 4852 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-home\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:55.729419 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.729454 4852 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-tmp-dir\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:55.729419 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.729470 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgpfr\" (UniqueName: \"kubernetes.io/projected/c69e53e0-661a-4a14-94de-d4450a554c80-kube-api-access-pgpfr\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:55.729974 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.729591 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-model-cache" pod "9211f266-8d05-4bd8-9947-7893a9c0634b" (UID: "9211f266-8d05-4bd8-9947-7893a9c0634b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:55.730539 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.730515 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-dshm" pod "c69e53e0-661a-4a14-94de-d4450a554c80" (UID: "c69e53e0-661a-4a14-94de-d4450a554c80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:55.730847 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.730832 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-dshm" pod "9211f266-8d05-4bd8-9947-7893a9c0634b" (UID: "9211f266-8d05-4bd8-9947-7893a9c0634b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:55.730945 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.730923 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9211f266-8d05-4bd8-9947-7893a9c0634b-kube-api-access-mv2cs" pod "9211f266-8d05-4bd8-9947-7893a9c0634b" (UID: "9211f266-8d05-4bd8-9947-7893a9c0634b"). InnerVolumeSpecName "kube-api-access-mv2cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:55.731204 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.731178 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9211f266-8d05-4bd8-9947-7893a9c0634b-tls-certs" pod "9211f266-8d05-4bd8-9947-7893a9c0634b" (UID: "9211f266-8d05-4bd8-9947-7893a9c0634b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:55.731650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.731628 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c69e53e0-661a-4a14-94de-d4450a554c80-tls-certs" pod "c69e53e0-661a-4a14-94de-d4450a554c80" (UID: "c69e53e0-661a-4a14-94de-d4450a554c80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:55.773507 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.773475 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:55.831480 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.831453 4852 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-model-cache\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:55.831480 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.831479 4852 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9211f266-8d05-4bd8-9947-7893a9c0634b-tls-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:55.831480 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.831488 4852 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9211f266-8d05-4bd8-9947-7893a9c0634b-dshm\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:55.831480 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.831496 4852 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c69e53e0-661a-4a14-94de-d4450a554c80-dshm\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:55.831771 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.831506 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv2cs\" (UniqueName: \"kubernetes.io/projected/9211f266-8d05-4bd8-9947-7893a9c0634b-kube-api-access-mv2cs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:55.831771 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:55.831516 4852 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c69e53e0-661a-4a14-94de-d4450a554c80-tls-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:56.045447 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:56.045416 4852 generic.go:356] "Generic (PLEG): container finished" podID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerID="781b20931150a22ce2e7ce31ae0710f45917118500c93e1e9c319c5b3881a48f" exitCode=0 Nov 04 01:33:56.045579 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:56.045461 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" Nov 04 01:33:56.045579 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:56.045494 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" event={"ID":"9211f266-8d05-4bd8-9947-7893a9c0634b","Type":"ContainerDied","Data":"781b20931150a22ce2e7ce31ae0710f45917118500c93e1e9c319c5b3881a48f"} Nov 04 01:33:56.045579 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:56.045534 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67" event={"ID":"9211f266-8d05-4bd8-9947-7893a9c0634b","Type":"ContainerDied","Data":"3e20701e1773346ef3e8c2ec2bdd0ed39cfdf1ae138a1b059dc5ce08984d4b87"} Nov 04 01:33:56.045579 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:56.045550 4852 scope.go:122] "RemoveContainer" containerID="781b20931150a22ce2e7ce31ae0710f45917118500c93e1e9c319c5b3881a48f" Nov 04 01:33:56.047393 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:56.047336 4852 generic.go:356] "Generic (PLEG): container finished" podID="c69e53e0-661a-4a14-94de-d4450a554c80" containerID="a5e4847f710c596a6473fff78d95944d7d4a81e35b0ea084d344b6e6830ff388" exitCode=0 Nov 04 01:33:56.047393 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:56.047389 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" Nov 04 01:33:56.047613 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:56.047450 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" event={"ID":"c69e53e0-661a-4a14-94de-d4450a554c80","Type":"ContainerDied","Data":"a5e4847f710c596a6473fff78d95944d7d4a81e35b0ea084d344b6e6830ff388"} Nov 04 01:33:56.047613 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:56.047481 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms" event={"ID":"c69e53e0-661a-4a14-94de-d4450a554c80","Type":"ContainerDied","Data":"969c47d7ec553c872c82db6103061d0b136673d218ace32abebe1d66013c411f"} Nov 04 01:33:56.055339 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:56.055321 4852 scope.go:122] "RemoveContainer" containerID="781b20931150a22ce2e7ce31ae0710f45917118500c93e1e9c319c5b3881a48f" Nov 04 01:33:56.055631 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:33:56.055613 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"781b20931150a22ce2e7ce31ae0710f45917118500c93e1e9c319c5b3881a48f\": container with ID starting with 781b20931150a22ce2e7ce31ae0710f45917118500c93e1e9c319c5b3881a48f not found: ID does not exist" containerID="781b20931150a22ce2e7ce31ae0710f45917118500c93e1e9c319c5b3881a48f" Nov 04 01:33:56.055700 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:56.055639 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"781b20931150a22ce2e7ce31ae0710f45917118500c93e1e9c319c5b3881a48f"} err="failed to get container status \"781b20931150a22ce2e7ce31ae0710f45917118500c93e1e9c319c5b3881a48f\": rpc error: code = NotFound desc = could not find container \"781b20931150a22ce2e7ce31ae0710f45917118500c93e1e9c319c5b3881a48f\": container with ID starting with 781b20931150a22ce2e7ce31ae0710f45917118500c93e1e9c319c5b3881a48f not found: ID does not exist" Nov 04 01:33:56.055700 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:56.055660 4852 scope.go:122] "RemoveContainer" containerID="a5e4847f710c596a6473fff78d95944d7d4a81e35b0ea084d344b6e6830ff388" Nov 04 01:33:56.064118 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:56.064103 4852 scope.go:122] "RemoveContainer" containerID="a5e4847f710c596a6473fff78d95944d7d4a81e35b0ea084d344b6e6830ff388" Nov 04 01:33:56.064357 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:33:56.064340 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5e4847f710c596a6473fff78d95944d7d4a81e35b0ea084d344b6e6830ff388\": container with ID starting with a5e4847f710c596a6473fff78d95944d7d4a81e35b0ea084d344b6e6830ff388 not found: ID does not exist" containerID="a5e4847f710c596a6473fff78d95944d7d4a81e35b0ea084d344b6e6830ff388" Nov 04 01:33:56.064401 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:56.064363 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5e4847f710c596a6473fff78d95944d7d4a81e35b0ea084d344b6e6830ff388"} err="failed to get container status \"a5e4847f710c596a6473fff78d95944d7d4a81e35b0ea084d344b6e6830ff388\": rpc error: code = NotFound desc = could not find container \"a5e4847f710c596a6473fff78d95944d7d4a81e35b0ea084d344b6e6830ff388\": container with ID starting with a5e4847f710c596a6473fff78d95944d7d4a81e35b0ea084d344b6e6830ff388 not found: ID does not exist" Nov 04 01:33:56.083144 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:56.083121 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67"] Nov 04 01:33:56.097663 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:56.097644 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/premium-w4-099dd9-kserve-59cc96f9d5-q5c67"] Nov 04 01:33:56.118227 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:56.118205 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms"] Nov 04 01:33:56.128743 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:56.128723 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/simulator-w4-099dd9-kserve-5fb6586d44-rpkms"] Nov 04 01:33:56.304925 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:56.304628 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:57.023592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:57.022714 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:57.775666 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:57.775627 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:57.928351 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:57.928302 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:57.995065 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:57.995037 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" path="/var/lib/kubelet/pods/9211f266-8d05-4bd8-9947-7893a9c0634b/volumes" Nov 04 01:33:57.995473 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:57.995460 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" path="/var/lib/kubelet/pods/c69e53e0-661a-4a14-94de-d4450a554c80/volumes" Nov 04 01:33:58.089988 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.089951 4852 generic.go:356] "Generic (PLEG): container finished" podID="852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0" containerID="db0fa2dbac755bdb6c9d80c39f7beff25a73ef974a24bffac39465f2db5f9391" exitCode=0 Nov 04 01:33:58.090339 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.090025 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" event={"ID":"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0","Type":"ContainerDied","Data":"db0fa2dbac755bdb6c9d80c39f7beff25a73ef974a24bffac39465f2db5f9391"} Nov 04 01:33:58.246918 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.246895 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:58.273875 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.273851 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-workload-certs\") pod \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " Nov 04 01:33:58.274856 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.273906 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-kube-api-access-twmpp\" (UniqueName: \"kubernetes.io/projected/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-kube-api-access-twmpp\") pod \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " Nov 04 01:33:58.274856 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.274071 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-data\") pod \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " Nov 04 01:33:58.274856 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.274140 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-workload-socket\") pod \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " Nov 04 01:33:58.274856 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.274140 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-workload-certs" pod "852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0" (UID: "852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:58.274856 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.274198 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-token\" (UniqueName: \"kubernetes.io/projected/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-token\") pod \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " Nov 04 01:33:58.274856 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.274257 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-envoy\") pod \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " Nov 04 01:33:58.274856 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.274295 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istiod-ca-cert\") pod \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " Nov 04 01:33:58.274856 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.274345 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-credential-socket\") pod \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " Nov 04 01:33:58.274856 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.274398 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-podinfo\") pod \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\" (UID: \"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0\") " Nov 04 01:33:58.274856 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.274771 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istiod-ca-cert" pod "852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0" (UID: "852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:58.276703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.275131 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:58.276703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.275151 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:58.276703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.275173 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-workload-socket" pod "852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0" (UID: "852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:58.276703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.275318 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-data" pod "852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0" (UID: "852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:58.276703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.275344 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-credential-socket" pod "852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0" (UID: "852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:58.278735 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.278709 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-kube-api-access-twmpp" pod "852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0" (UID: "852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0"). InnerVolumeSpecName "kube-api-access-twmpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:58.278735 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.278723 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-envoy" pod "852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0" (UID: "852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:58.278886 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.278727 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-token" pod "852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0" (UID: "852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:58.278886 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.278780 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-podinfo" pod "852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0" (UID: "852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:58.380687 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.376757 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:58.380687 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.376794 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:58.380687 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.376809 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:58.380687 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.376821 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:58.380687 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.376835 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:58.380687 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.376848 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:58.380687 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.376861 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twmpp\" (UniqueName: \"kubernetes.io/projected/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0-kube-api-access-twmpp\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:58.434556 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.434516 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:58.714596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.714482 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:58.905549 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:58.905402 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:59.100476 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:59.100398 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" event={"ID":"852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0","Type":"ContainerDied","Data":"f88637aec3e9c996b416e81645165a0aa9398e6a2292862c46163a1117a13bf3"} Nov 04 01:33:59.100476 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:59.100478 4852 scope.go:122] "RemoveContainer" containerID="db0fa2dbac755bdb6c9d80c39f7beff25a73ef974a24bffac39465f2db5f9391" Nov 04 01:33:59.101001 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:59.100679 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9" Nov 04 01:33:59.146707 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:59.146680 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9"] Nov 04 01:33:59.154588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:59.154538 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-mh2b9"] Nov 04 01:33:59.254058 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:59.254018 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:59.499115 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:59.498907 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:59.682459 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:33:59.682414 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:33:59.866366 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:33:59.866332 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:34:00.032589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:00.031612 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0" path="/var/lib/kubelet/pods/852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0/volumes" Nov 04 01:34:00.061499 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:00.061459 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:00.965114 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:00.961846 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:01.132190 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:01.132144 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:01.646589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:01.644402 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:02.335375 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:02.325843 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:02.472864 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:02.472835 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:02.964655 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:02.963318 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:03.617363 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:03.615939 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:04.654594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:04.654533 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:05.197808 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:05.197757 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:05.484352 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:05.484268 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9"] Nov 04 01:34:05.605794 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:05.605757 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:05.748317 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:05.748228 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:05.778125 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:05.778091 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb"] Nov 04 01:34:05.778601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:05.778541 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" podUID="7d5e42b9-cad8-4145-8824-f972cc7d6505" containerName="maas-api" containerID="cri-o://70f4abf51b789c0b945c613e1d490dd3d5d957b6108f7e58d52afecec81b13e0" gracePeriod=30 Nov 04 01:34:05.892223 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:05.892182 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:06.088286 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.088258 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" Nov 04 01:34:06.461963 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.461869 4852 generic.go:356] "Generic (PLEG): container finished" podID="7d5e42b9-cad8-4145-8824-f972cc7d6505" containerID="70f4abf51b789c0b945c613e1d490dd3d5d957b6108f7e58d52afecec81b13e0" exitCode=0 Nov 04 01:34:06.462397 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.462371 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" event={"ID":"7d5e42b9-cad8-4145-8824-f972cc7d6505","Type":"ContainerDied","Data":"70f4abf51b789c0b945c613e1d490dd3d5d957b6108f7e58d52afecec81b13e0"} Nov 04 01:34:06.462542 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.462445 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" event={"ID":"7d5e42b9-cad8-4145-8824-f972cc7d6505","Type":"ContainerDied","Data":"1ac7693dfed7e7d15c1b161ae0875e5dde7e86e0f3d96b2f2d1b64ccbc98f769"} Nov 04 01:34:06.462542 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.462519 4852 scope.go:122] "RemoveContainer" containerID="70f4abf51b789c0b945c613e1d490dd3d5d957b6108f7e58d52afecec81b13e0" Nov 04 01:34:06.462830 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.462813 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb" Nov 04 01:34:06.463354 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.463329 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:06.466260 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.466236 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5e42b9-cad8-4145-8824-f972cc7d6505-kube-api-access-gxgwc\" (UniqueName: \"kubernetes.io/projected/7d5e42b9-cad8-4145-8824-f972cc7d6505-kube-api-access-gxgwc\") pod \"7d5e42b9-cad8-4145-8824-f972cc7d6505\" (UID: \"7d5e42b9-cad8-4145-8824-f972cc7d6505\") " Nov 04 01:34:06.466421 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.466400 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d5e42b9-cad8-4145-8824-f972cc7d6505-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d5e42b9-cad8-4145-8824-f972cc7d6505-metrics-certs\") pod \"7d5e42b9-cad8-4145-8824-f972cc7d6505\" (UID: \"7d5e42b9-cad8-4145-8824-f972cc7d6505\") " Nov 04 01:34:06.466543 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.466529 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d5e42b9-cad8-4145-8824-f972cc7d6505-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d5e42b9-cad8-4145-8824-f972cc7d6505-openshift-service-ca\") pod \"7d5e42b9-cad8-4145-8824-f972cc7d6505\" (UID: \"7d5e42b9-cad8-4145-8824-f972cc7d6505\") " Nov 04 01:34:06.466673 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.466658 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d5e42b9-cad8-4145-8824-f972cc7d6505-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d5e42b9-cad8-4145-8824-f972cc7d6505-maas-api-tls\") pod \"7d5e42b9-cad8-4145-8824-f972cc7d6505\" (UID: \"7d5e42b9-cad8-4145-8824-f972cc7d6505\") " Nov 04 01:34:06.479597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.474885 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5e42b9-cad8-4145-8824-f972cc7d6505-openshift-service-ca" pod "7d5e42b9-cad8-4145-8824-f972cc7d6505" (UID: "7d5e42b9-cad8-4145-8824-f972cc7d6505"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:06.539690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.539659 4852 scope.go:122] "RemoveContainer" containerID="70f4abf51b789c0b945c613e1d490dd3d5d957b6108f7e58d52afecec81b13e0" Nov 04 01:34:06.547650 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:34:06.540085 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f4abf51b789c0b945c613e1d490dd3d5d957b6108f7e58d52afecec81b13e0\": container with ID starting with 70f4abf51b789c0b945c613e1d490dd3d5d957b6108f7e58d52afecec81b13e0 not found: ID does not exist" containerID="70f4abf51b789c0b945c613e1d490dd3d5d957b6108f7e58d52afecec81b13e0" Nov 04 01:34:06.547650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.540125 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f4abf51b789c0b945c613e1d490dd3d5d957b6108f7e58d52afecec81b13e0"} err="failed to get container status \"70f4abf51b789c0b945c613e1d490dd3d5d957b6108f7e58d52afecec81b13e0\": rpc error: code = NotFound desc = could not find container \"70f4abf51b789c0b945c613e1d490dd3d5d957b6108f7e58d52afecec81b13e0\": container with ID starting with 70f4abf51b789c0b945c613e1d490dd3d5d957b6108f7e58d52afecec81b13e0 not found: ID does not exist" Nov 04 01:34:06.607022 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.606984 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d5e42b9-cad8-4145-8824-f972cc7d6505-maas-api-tls" pod "7d5e42b9-cad8-4145-8824-f972cc7d6505" (UID: "7d5e42b9-cad8-4145-8824-f972cc7d6505"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:06.608552 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.608522 4852 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d5e42b9-cad8-4145-8824-f972cc7d6505-openshift-service-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:06.608552 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.608552 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d5e42b9-cad8-4145-8824-f972cc7d6505-maas-api-tls\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:06.611453 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.611426 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d5e42b9-cad8-4145-8824-f972cc7d6505-metrics-certs" pod "7d5e42b9-cad8-4145-8824-f972cc7d6505" (UID: "7d5e42b9-cad8-4145-8824-f972cc7d6505"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:06.611602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.611470 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5e42b9-cad8-4145-8824-f972cc7d6505-kube-api-access-gxgwc" pod "7d5e42b9-cad8-4145-8824-f972cc7d6505" (UID: "7d5e42b9-cad8-4145-8824-f972cc7d6505"). InnerVolumeSpecName "kube-api-access-gxgwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:06.658017 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.657994 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:06.710242 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.710214 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxgwc\" (UniqueName: \"kubernetes.io/projected/7d5e42b9-cad8-4145-8824-f972cc7d6505-kube-api-access-gxgwc\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:06.710242 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.710241 4852 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d5e42b9-cad8-4145-8824-f972cc7d6505-metrics-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:06.798847 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.798816 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb"] Nov 04 01:34:06.801882 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:06.801859 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8e9791-6969d8c7bf-fxmsb"] Nov 04 01:34:07.501588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:07.500928 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:07.682879 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:07.682840 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:07.996363 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:07.996331 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d5e42b9-cad8-4145-8824-f972cc7d6505" path="/var/lib/kubelet/pods/7d5e42b9-cad8-4145-8824-f972cc7d6505/volumes" Nov 04 01:34:08.581590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:08.580987 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:09.213409 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:09.213359 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:09.441466 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:09.441435 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:09.783147 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:09.783052 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:10.014396 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:10.014364 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk"] Nov 04 01:34:10.014833 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:10.014803 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" podUID="6604612f-7353-40ab-8a1b-29b19b3395f0" containerName="istio-proxy" containerID="cri-o://2683c5f12c7114cd0d61b5b8aafe42e5cc34403fcf1618a30ce756202182f4ff" gracePeriod=30 Nov 04 01:34:10.037955 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:10.037829 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:10.234587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:10.229271 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:10.484958 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:10.484844 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9" podUID="f3ab7ef8-0104-4e1f-8327-41964669c3b3" containerName="payload-pre-processing" containerID="cri-o://79820189a70b7167614eb3153c36a1eaf94bb1834ef31df1e257b37bce3106e4" gracePeriod=25 Nov 04 01:34:10.553954 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:10.553918 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:10.680668 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:10.680489 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" containerID="cri-o://8cfc22a3a39a4a06c6498abbbec07f492d4961501df585f5f816bfe7674349dc" gracePeriod=45 Nov 04 01:34:10.789883 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:10.789841 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": dial tcp 10.134.0.36:8000: connect: connection refused" Nov 04 01:34:10.820931 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:10.820904 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9" Nov 04 01:34:11.011532 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.011501 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:34:11.018316 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.018283 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3ab7ef8-0104-4e1f-8327-41964669c3b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3ab7ef8-0104-4e1f-8327-41964669c3b3-plugins-config-volume\") pod \"f3ab7ef8-0104-4e1f-8327-41964669c3b3\" (UID: \"f3ab7ef8-0104-4e1f-8327-41964669c3b3\") " Nov 04 01:34:11.018485 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.018467 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3ab7ef8-0104-4e1f-8327-41964669c3b3-kube-api-access-ldcxv\" (UniqueName: \"kubernetes.io/projected/f3ab7ef8-0104-4e1f-8327-41964669c3b3-kube-api-access-ldcxv\") pod \"f3ab7ef8-0104-4e1f-8327-41964669c3b3\" (UID: \"f3ab7ef8-0104-4e1f-8327-41964669c3b3\") " Nov 04 01:34:11.018815 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.018788 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3ab7ef8-0104-4e1f-8327-41964669c3b3-plugins-config-volume" pod "f3ab7ef8-0104-4e1f-8327-41964669c3b3" (UID: "f3ab7ef8-0104-4e1f-8327-41964669c3b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:11.020339 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.020316 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3ab7ef8-0104-4e1f-8327-41964669c3b3-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.022757 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.021778 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ab7ef8-0104-4e1f-8327-41964669c3b3-kube-api-access-ldcxv" pod "f3ab7ef8-0104-4e1f-8327-41964669c3b3" (UID: "f3ab7ef8-0104-4e1f-8327-41964669c3b3"). InnerVolumeSpecName "kube-api-access-ldcxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:11.121624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.121519 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ce1f2a5-3954-4044-a85e-2761a6b2088e-tls-certs\" (UniqueName: \"kubernetes.io/secret/1ce1f2a5-3954-4044-a85e-2761a6b2088e-tls-certs\") pod \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " Nov 04 01:34:11.121857 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.121839 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-tmp-dir\") pod \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " Nov 04 01:34:11.121990 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.121972 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-home\" (UniqueName: \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-home\") pod \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " Nov 04 01:34:11.122145 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.122131 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-dshm\") pod \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " Nov 04 01:34:11.122278 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.122264 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-model-cache\") pod \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " Nov 04 01:34:11.122504 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.122487 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce1f2a5-3954-4044-a85e-2761a6b2088e-kube-api-access-lnddd\" (UniqueName: \"kubernetes.io/projected/1ce1f2a5-3954-4044-a85e-2761a6b2088e-kube-api-access-lnddd\") pod \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\" (UID: \"1ce1f2a5-3954-4044-a85e-2761a6b2088e\") " Nov 04 01:34:11.123520 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.123499 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldcxv\" (UniqueName: \"kubernetes.io/projected/f3ab7ef8-0104-4e1f-8327-41964669c3b3-kube-api-access-ldcxv\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.125411 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.125383 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-tmp-dir" pod "1ce1f2a5-3954-4044-a85e-2761a6b2088e" (UID: "1ce1f2a5-3954-4044-a85e-2761a6b2088e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:11.125579 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.125541 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-home" pod "1ce1f2a5-3954-4044-a85e-2761a6b2088e" (UID: "1ce1f2a5-3954-4044-a85e-2761a6b2088e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:11.125650 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.125601 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-model-cache" pod "1ce1f2a5-3954-4044-a85e-2761a6b2088e" (UID: "1ce1f2a5-3954-4044-a85e-2761a6b2088e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:11.126715 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.126679 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ce1f2a5-3954-4044-a85e-2761a6b2088e-tls-certs" pod "1ce1f2a5-3954-4044-a85e-2761a6b2088e" (UID: "1ce1f2a5-3954-4044-a85e-2761a6b2088e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:11.127633 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.127599 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce1f2a5-3954-4044-a85e-2761a6b2088e-kube-api-access-lnddd" pod "1ce1f2a5-3954-4044-a85e-2761a6b2088e" (UID: "1ce1f2a5-3954-4044-a85e-2761a6b2088e"). InnerVolumeSpecName "kube-api-access-lnddd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:11.128040 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.128013 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-dshm" pod "1ce1f2a5-3954-4044-a85e-2761a6b2088e" (UID: "1ce1f2a5-3954-4044-a85e-2761a6b2088e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:11.182584 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.182523 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:11.240604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.239644 4852 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ce1f2a5-3954-4044-a85e-2761a6b2088e-tls-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.240604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.239684 4852 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-tmp-dir\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.240604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.239700 4852 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-home\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.240604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.239712 4852 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-dshm\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.240604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.239726 4852 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ce1f2a5-3954-4044-a85e-2761a6b2088e-model-cache\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.240604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.239742 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnddd\" (UniqueName: \"kubernetes.io/projected/1ce1f2a5-3954-4044-a85e-2761a6b2088e-kube-api-access-lnddd\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.535667 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.535631 4852 generic.go:356] "Generic (PLEG): container finished" podID="f3ab7ef8-0104-4e1f-8327-41964669c3b3" containerID="79820189a70b7167614eb3153c36a1eaf94bb1834ef31df1e257b37bce3106e4" exitCode=0 Nov 04 01:34:11.535812 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.535700 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9" Nov 04 01:34:11.535812 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.535711 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9" event={"ID":"f3ab7ef8-0104-4e1f-8327-41964669c3b3","Type":"ContainerDied","Data":"79820189a70b7167614eb3153c36a1eaf94bb1834ef31df1e257b37bce3106e4"} Nov 04 01:34:11.535812 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.535762 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9" event={"ID":"f3ab7ef8-0104-4e1f-8327-41964669c3b3","Type":"ContainerDied","Data":"8119424db22250b2054357f70df9c305418f7f03f163f98a512e30053c727380"} Nov 04 01:34:11.535812 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.535787 4852 scope.go:122] "RemoveContainer" containerID="79820189a70b7167614eb3153c36a1eaf94bb1834ef31df1e257b37bce3106e4" Nov 04 01:34:11.537271 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.537250 4852 generic.go:356] "Generic (PLEG): container finished" podID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerID="8cfc22a3a39a4a06c6498abbbec07f492d4961501df585f5f816bfe7674349dc" exitCode=0 Nov 04 01:34:11.537376 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.537312 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" Nov 04 01:34:11.537457 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.537344 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" event={"ID":"1ce1f2a5-3954-4044-a85e-2761a6b2088e","Type":"ContainerDied","Data":"8cfc22a3a39a4a06c6498abbbec07f492d4961501df585f5f816bfe7674349dc"} Nov 04 01:34:11.537534 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.537484 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6" event={"ID":"1ce1f2a5-3954-4044-a85e-2761a6b2088e","Type":"ContainerDied","Data":"79d0fed5bb2fe15594bb27cfbb914c57789b21aea54182302740e78991df0e21"} Nov 04 01:34:11.547713 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.547452 4852 scope.go:122] "RemoveContainer" containerID="79820189a70b7167614eb3153c36a1eaf94bb1834ef31df1e257b37bce3106e4" Nov 04 01:34:11.547784 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:34:11.547765 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79820189a70b7167614eb3153c36a1eaf94bb1834ef31df1e257b37bce3106e4\": container with ID starting with 79820189a70b7167614eb3153c36a1eaf94bb1834ef31df1e257b37bce3106e4 not found: ID does not exist" containerID="79820189a70b7167614eb3153c36a1eaf94bb1834ef31df1e257b37bce3106e4" Nov 04 01:34:11.547831 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.547793 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79820189a70b7167614eb3153c36a1eaf94bb1834ef31df1e257b37bce3106e4"} err="failed to get container status \"79820189a70b7167614eb3153c36a1eaf94bb1834ef31df1e257b37bce3106e4\": rpc error: code = NotFound desc = could not find container \"79820189a70b7167614eb3153c36a1eaf94bb1834ef31df1e257b37bce3106e4\": container with ID starting with 79820189a70b7167614eb3153c36a1eaf94bb1834ef31df1e257b37bce3106e4 not found: ID does not exist" Nov 04 01:34:11.547831 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.547811 4852 scope.go:122] "RemoveContainer" containerID="8cfc22a3a39a4a06c6498abbbec07f492d4961501df585f5f816bfe7674349dc" Nov 04 01:34:11.581839 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.581814 4852 scope.go:122] "RemoveContainer" containerID="8cfc22a3a39a4a06c6498abbbec07f492d4961501df585f5f816bfe7674349dc" Nov 04 01:34:11.582145 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:34:11.582124 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cfc22a3a39a4a06c6498abbbec07f492d4961501df585f5f816bfe7674349dc\": container with ID starting with 8cfc22a3a39a4a06c6498abbbec07f492d4961501df585f5f816bfe7674349dc not found: ID does not exist" containerID="8cfc22a3a39a4a06c6498abbbec07f492d4961501df585f5f816bfe7674349dc" Nov 04 01:34:11.582243 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.582158 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cfc22a3a39a4a06c6498abbbec07f492d4961501df585f5f816bfe7674349dc"} err="failed to get container status \"8cfc22a3a39a4a06c6498abbbec07f492d4961501df585f5f816bfe7674349dc\": rpc error: code = NotFound desc = could not find container \"8cfc22a3a39a4a06c6498abbbec07f492d4961501df585f5f816bfe7674349dc\": container with ID starting with 8cfc22a3a39a4a06c6498abbbec07f492d4961501df585f5f816bfe7674349dc not found: ID does not exist" Nov 04 01:34:11.588505 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.588481 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6"] Nov 04 01:34:11.594817 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.594790 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/premium-w5-9c8bf5-kserve-7775cfd4d8-5hkw6"] Nov 04 01:34:11.607015 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.606996 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9"] Nov 04 01:34:11.611385 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.611366 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fff32-6c97457c55-l4hr9"] Nov 04 01:34:11.914194 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:11.914160 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:12.002280 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:12.002247 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" path="/var/lib/kubelet/pods/1ce1f2a5-3954-4044-a85e-2761a6b2088e/volumes" Nov 04 01:34:12.002778 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:12.002762 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3ab7ef8-0104-4e1f-8327-41964669c3b3" path="/var/lib/kubelet/pods/f3ab7ef8-0104-4e1f-8327-41964669c3b3/volumes" Nov 04 01:34:12.426609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:12.426124 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:12.630636 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:12.630602 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:34:12.878688 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:12.878657 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:13.098711 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:13.098625 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:13.630415 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:13.630380 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:14.551025 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:14.550991 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:14.914642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:14.913493 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:15.380609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.375547 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:34:15.429278 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.428065 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6604612f-7353-40ab-8a1b-29b19b3395f0-kube-api-access-t46jl\" (UniqueName: \"kubernetes.io/projected/6604612f-7353-40ab-8a1b-29b19b3395f0-kube-api-access-t46jl\") pod \"6604612f-7353-40ab-8a1b-29b19b3395f0\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " Nov 04 01:34:15.429278 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.428125 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-token\" (UniqueName: \"kubernetes.io/projected/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-token\") pod \"6604612f-7353-40ab-8a1b-29b19b3395f0\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " Nov 04 01:34:15.429278 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.428197 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-data\") pod \"6604612f-7353-40ab-8a1b-29b19b3395f0\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " Nov 04 01:34:15.429278 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.428275 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6604612f-7353-40ab-8a1b-29b19b3395f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6604612f-7353-40ab-8a1b-29b19b3395f0-istiod-ca-cert\") pod \"6604612f-7353-40ab-8a1b-29b19b3395f0\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " Nov 04 01:34:15.429278 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.428300 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-podinfo\") pod \"6604612f-7353-40ab-8a1b-29b19b3395f0\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " Nov 04 01:34:15.429278 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.428364 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-workload-socket\") pod \"6604612f-7353-40ab-8a1b-29b19b3395f0\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " Nov 04 01:34:15.429278 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.428382 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-credential-socket\") pod \"6604612f-7353-40ab-8a1b-29b19b3395f0\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " Nov 04 01:34:15.429278 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.428426 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-workload-certs\") pod \"6604612f-7353-40ab-8a1b-29b19b3395f0\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " Nov 04 01:34:15.429278 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.428457 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-envoy\") pod \"6604612f-7353-40ab-8a1b-29b19b3395f0\" (UID: \"6604612f-7353-40ab-8a1b-29b19b3395f0\") " Nov 04 01:34:15.437797 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.432264 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-data" pod "6604612f-7353-40ab-8a1b-29b19b3395f0" (UID: "6604612f-7353-40ab-8a1b-29b19b3395f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:15.437797 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.432319 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-workload-socket" pod "6604612f-7353-40ab-8a1b-29b19b3395f0" (UID: "6604612f-7353-40ab-8a1b-29b19b3395f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:15.437797 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.432357 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-workload-certs" pod "6604612f-7353-40ab-8a1b-29b19b3395f0" (UID: "6604612f-7353-40ab-8a1b-29b19b3395f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:15.437797 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.432396 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-credential-socket" pod "6604612f-7353-40ab-8a1b-29b19b3395f0" (UID: "6604612f-7353-40ab-8a1b-29b19b3395f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:15.437797 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.432505 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6604612f-7353-40ab-8a1b-29b19b3395f0-istiod-ca-cert" pod "6604612f-7353-40ab-8a1b-29b19b3395f0" (UID: "6604612f-7353-40ab-8a1b-29b19b3395f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:15.437797 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.436685 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-envoy" pod "6604612f-7353-40ab-8a1b-29b19b3395f0" (UID: "6604612f-7353-40ab-8a1b-29b19b3395f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:15.440840 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.440753 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-token" pod "6604612f-7353-40ab-8a1b-29b19b3395f0" (UID: "6604612f-7353-40ab-8a1b-29b19b3395f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:15.441700 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.441041 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-podinfo" pod "6604612f-7353-40ab-8a1b-29b19b3395f0" (UID: "6604612f-7353-40ab-8a1b-29b19b3395f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:15.443248 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.443192 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6604612f-7353-40ab-8a1b-29b19b3395f0-kube-api-access-t46jl" pod "6604612f-7353-40ab-8a1b-29b19b3395f0" (UID: "6604612f-7353-40ab-8a1b-29b19b3395f0"). InnerVolumeSpecName "kube-api-access-t46jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:15.480898 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.480865 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq"] Nov 04 01:34:15.530350 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.530314 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:15.530350 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.530344 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:15.530350 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.530354 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:15.530627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.530362 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:15.530627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.530371 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t46jl\" (UniqueName: \"kubernetes.io/projected/6604612f-7353-40ab-8a1b-29b19b3395f0-kube-api-access-t46jl\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:15.530627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.530380 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:15.530627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.530388 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:15.530627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.530397 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6604612f-7353-40ab-8a1b-29b19b3395f0-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:15.530627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.530404 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6604612f-7353-40ab-8a1b-29b19b3395f0-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:15.626600 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.626528 4852 generic.go:356] "Generic (PLEG): container finished" podID="6604612f-7353-40ab-8a1b-29b19b3395f0" containerID="2683c5f12c7114cd0d61b5b8aafe42e5cc34403fcf1618a30ce756202182f4ff" exitCode=0 Nov 04 01:34:15.626600 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.626587 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" event={"ID":"6604612f-7353-40ab-8a1b-29b19b3395f0","Type":"ContainerDied","Data":"2683c5f12c7114cd0d61b5b8aafe42e5cc34403fcf1618a30ce756202182f4ff"} Nov 04 01:34:15.626907 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.626633 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" Nov 04 01:34:15.626907 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.626686 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk" event={"ID":"6604612f-7353-40ab-8a1b-29b19b3395f0","Type":"ContainerDied","Data":"16835452ef1317104f7ed86be665878e979ce729415922e9e26ad28124d8b1da"} Nov 04 01:34:15.626907 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.626718 4852 scope.go:122] "RemoveContainer" containerID="2683c5f12c7114cd0d61b5b8aafe42e5cc34403fcf1618a30ce756202182f4ff" Nov 04 01:34:15.638881 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.638862 4852 scope.go:122] "RemoveContainer" containerID="61ba8bb1d7fb264cb90f65de8d21537ccc1d63ee031575b78765002f4e599288" Nov 04 01:34:15.648984 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.648966 4852 scope.go:122] "RemoveContainer" containerID="2683c5f12c7114cd0d61b5b8aafe42e5cc34403fcf1618a30ce756202182f4ff" Nov 04 01:34:15.649244 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:34:15.649226 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2683c5f12c7114cd0d61b5b8aafe42e5cc34403fcf1618a30ce756202182f4ff\": container with ID starting with 2683c5f12c7114cd0d61b5b8aafe42e5cc34403fcf1618a30ce756202182f4ff not found: ID does not exist" containerID="2683c5f12c7114cd0d61b5b8aafe42e5cc34403fcf1618a30ce756202182f4ff" Nov 04 01:34:15.649330 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.649252 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2683c5f12c7114cd0d61b5b8aafe42e5cc34403fcf1618a30ce756202182f4ff"} err="failed to get container status \"2683c5f12c7114cd0d61b5b8aafe42e5cc34403fcf1618a30ce756202182f4ff\": rpc error: code = NotFound desc = could not find container \"2683c5f12c7114cd0d61b5b8aafe42e5cc34403fcf1618a30ce756202182f4ff\": container with ID starting with 2683c5f12c7114cd0d61b5b8aafe42e5cc34403fcf1618a30ce756202182f4ff not found: ID does not exist" Nov 04 01:34:15.649330 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.649267 4852 scope.go:122] "RemoveContainer" containerID="61ba8bb1d7fb264cb90f65de8d21537ccc1d63ee031575b78765002f4e599288" Nov 04 01:34:15.649489 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:34:15.649470 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61ba8bb1d7fb264cb90f65de8d21537ccc1d63ee031575b78765002f4e599288\": container with ID starting with 61ba8bb1d7fb264cb90f65de8d21537ccc1d63ee031575b78765002f4e599288 not found: ID does not exist" containerID="61ba8bb1d7fb264cb90f65de8d21537ccc1d63ee031575b78765002f4e599288" Nov 04 01:34:15.649550 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.649492 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61ba8bb1d7fb264cb90f65de8d21537ccc1d63ee031575b78765002f4e599288"} err="failed to get container status \"61ba8bb1d7fb264cb90f65de8d21537ccc1d63ee031575b78765002f4e599288\": rpc error: code = NotFound desc = could not find container \"61ba8bb1d7fb264cb90f65de8d21537ccc1d63ee031575b78765002f4e599288\": container with ID starting with 61ba8bb1d7fb264cb90f65de8d21537ccc1d63ee031575b78765002f4e599288 not found: ID does not exist" Nov 04 01:34:15.654500 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.654479 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:15.672528 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.672504 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk"] Nov 04 01:34:15.693333 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.693311 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8e9791-openshift-default-7f6f76964f-rngkk"] Nov 04 01:34:15.889132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.889038 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:15.995883 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:15.995854 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6604612f-7353-40ab-8a1b-29b19b3395f0" path="/var/lib/kubelet/pods/6604612f-7353-40ab-8a1b-29b19b3395f0/volumes" Nov 04 01:34:16.267169 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:16.267139 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:16.632181 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:16.632099 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:16.806660 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:16.806506 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:16.971241 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:16.970051 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:17.753232 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:17.749109 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:18.243065 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:18.243026 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:18.399467 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:18.399346 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:18.571242 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:18.571191 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:18.660361 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:18.659500 4852 generic.go:356] "Generic (PLEG): container finished" podID="1efe83d4-74d6-431e-91fd-314ce0f5d116" containerID="392fea8c004cca56e05fd1f56f459bc5f07ec94d8d4b6daf9bb2fb2fb7596c91" exitCode=0 Nov 04 01:34:18.660361 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:18.659709 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" event={"ID":"1efe83d4-74d6-431e-91fd-314ce0f5d116","Type":"ContainerDied","Data":"392fea8c004cca56e05fd1f56f459bc5f07ec94d8d4b6daf9bb2fb2fb7596c91"} Nov 04 01:34:18.660361 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:18.660243 4852 scope.go:122] "RemoveContainer" containerID="392fea8c004cca56e05fd1f56f459bc5f07ec94d8d4b6daf9bb2fb2fb7596c91" Nov 04 01:34:18.662300 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:18.662267 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:34:18.662416 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:18.662347 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:34:18.662416 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:18.662395 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:34:18.772176 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:18.772147 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:19.090059 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:19.090029 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p"] Nov 04 01:34:19.090258 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:19.090235 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" podUID="baefa3c0-28be-428a-9591-f0ca1f07fafa" containerName="istio-proxy" containerID="cri-o://b8bce3eaa168e4e58088218699c821309762b5813ac17399f4653bec114d0e74" gracePeriod=30 Nov 04 01:34:19.113746 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:19.113704 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw"] Nov 04 01:34:19.114052 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:19.114012 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" podUID="ab9122ac-8874-4a64-9137-a7fe056db11e" containerName="istio-proxy" containerID="cri-o://341c1ef2ada34d68609175467758b55c188435bbdf83814e34ab1401e6866575" gracePeriod=30 Nov 04 01:34:19.247629 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:19.247585 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:19.652776 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:19.652742 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:19.684587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:19.679878 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" event={"ID":"1efe83d4-74d6-431e-91fd-314ce0f5d116","Type":"ContainerStarted","Data":"e5c328e2adcecb5bdf01a8f879927b213e733e73e6ec04e0b543fba2c2491d28"} Nov 04 01:34:19.908649 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:19.905805 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:34:19.915587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:19.913720 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 04 01:34:19.915587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:19.913796 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" podUID="1efe83d4-74d6-431e-91fd-314ce0f5d116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 04 01:34:20.288180 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:20.288130 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:20.531879 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:20.531843 4852 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:34:20.532054 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:20.531916 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" podUID="baefa3c0-28be-428a-9591-f0ca1f07fafa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:34:20.884590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:20.880180 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:20.911986 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:20.911949 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 04 01:34:20.912642 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:20.912538 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" podUID="1efe83d4-74d6-431e-91fd-314ce0f5d116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 04 01:34:21.526890 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:21.526857 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6"] Nov 04 01:34:21.697062 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:21.696996 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:21.908062 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:21.907965 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 04 01:34:21.908458 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:21.908410 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" podUID="1efe83d4-74d6-431e-91fd-314ce0f5d116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 04 01:34:21.942267 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:21.942223 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:22.229263 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:22.229156 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:22.905719 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:22.905688 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 04 01:34:22.905925 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:22.905757 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" podUID="1efe83d4-74d6-431e-91fd-314ce0f5d116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 04 01:34:22.947282 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:22.947248 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:34:23.143258 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:23.143214 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:23.905969 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:23.905936 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 04 01:34:23.906142 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:23.905989 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" podUID="1efe83d4-74d6-431e-91fd-314ce0f5d116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 04 01:34:24.107156 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.107114 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:24.293395 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.293305 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:24.485286 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.484820 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:34:24.509379 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.509359 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:34:24.551089 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.551045 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-podinfo\") pod \"baefa3c0-28be-428a-9591-f0ca1f07fafa\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " Nov 04 01:34:24.551255 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.551095 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-credential-socket\") pod \"ab9122ac-8874-4a64-9137-a7fe056db11e\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " Nov 04 01:34:24.551255 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.551159 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-workload-socket\") pod \"ab9122ac-8874-4a64-9137-a7fe056db11e\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " Nov 04 01:34:24.551255 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.551187 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-data\") pod \"ab9122ac-8874-4a64-9137-a7fe056db11e\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " Nov 04 01:34:24.551255 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.551222 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-envoy\") pod \"ab9122ac-8874-4a64-9137-a7fe056db11e\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " Nov 04 01:34:24.551255 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.551249 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-envoy\") pod \"baefa3c0-28be-428a-9591-f0ca1f07fafa\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " Nov 04 01:34:24.551540 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.551280 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-workload-certs\") pod \"baefa3c0-28be-428a-9591-f0ca1f07fafa\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " Nov 04 01:34:24.551540 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.551348 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-credential-socket" pod "ab9122ac-8874-4a64-9137-a7fe056db11e" (UID: "ab9122ac-8874-4a64-9137-a7fe056db11e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.551540 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.551428 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-workload-socket" pod "ab9122ac-8874-4a64-9137-a7fe056db11e" (UID: "ab9122ac-8874-4a64-9137-a7fe056db11e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.551730 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.551647 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-workload-certs" pod "baefa3c0-28be-428a-9591-f0ca1f07fafa" (UID: "baefa3c0-28be-428a-9591-f0ca1f07fafa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.551730 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.551691 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-credential-socket\") pod \"baefa3c0-28be-428a-9591-f0ca1f07fafa\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " Nov 04 01:34:24.551831 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.551758 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-workload-socket\") pod \"baefa3c0-28be-428a-9591-f0ca1f07fafa\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " Nov 04 01:34:24.551831 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.551817 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baefa3c0-28be-428a-9591-f0ca1f07fafa-kube-api-access-wkcjp\" (UniqueName: \"kubernetes.io/projected/baefa3c0-28be-428a-9591-f0ca1f07fafa-kube-api-access-wkcjp\") pod \"baefa3c0-28be-428a-9591-f0ca1f07fafa\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " Nov 04 01:34:24.551940 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.551859 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-workload-certs\") pod \"ab9122ac-8874-4a64-9137-a7fe056db11e\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " Nov 04 01:34:24.551940 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.551889 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-credential-socket" pod "baefa3c0-28be-428a-9591-f0ca1f07fafa" (UID: "baefa3c0-28be-428a-9591-f0ca1f07fafa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.552048 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.552014 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-workload-socket" pod "baefa3c0-28be-428a-9591-f0ca1f07fafa" (UID: "baefa3c0-28be-428a-9591-f0ca1f07fafa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.552091 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.552074 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-workload-certs" pod "ab9122ac-8874-4a64-9137-a7fe056db11e" (UID: "ab9122ac-8874-4a64-9137-a7fe056db11e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.552320 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.552292 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-token\" (UniqueName: \"kubernetes.io/projected/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-token\") pod \"ab9122ac-8874-4a64-9137-a7fe056db11e\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " Nov 04 01:34:24.552415 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.552356 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-token\" (UniqueName: \"kubernetes.io/projected/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-token\") pod \"baefa3c0-28be-428a-9591-f0ca1f07fafa\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " Nov 04 01:34:24.552415 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.552391 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab9122ac-8874-4a64-9137-a7fe056db11e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9122ac-8874-4a64-9137-a7fe056db11e-istiod-ca-cert\") pod \"ab9122ac-8874-4a64-9137-a7fe056db11e\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " Nov 04 01:34:24.552748 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.552424 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab9122ac-8874-4a64-9137-a7fe056db11e-kube-api-access-79qks\" (UniqueName: \"kubernetes.io/projected/ab9122ac-8874-4a64-9137-a7fe056db11e-kube-api-access-79qks\") pod \"ab9122ac-8874-4a64-9137-a7fe056db11e\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " Nov 04 01:34:24.552748 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.552529 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baefa3c0-28be-428a-9591-f0ca1f07fafa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baefa3c0-28be-428a-9591-f0ca1f07fafa-istiod-ca-cert\") pod \"baefa3c0-28be-428a-9591-f0ca1f07fafa\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " Nov 04 01:34:24.552748 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.552617 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-podinfo\") pod \"ab9122ac-8874-4a64-9137-a7fe056db11e\" (UID: \"ab9122ac-8874-4a64-9137-a7fe056db11e\") " Nov 04 01:34:24.552921 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.552863 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-data" pod "ab9122ac-8874-4a64-9137-a7fe056db11e" (UID: "ab9122ac-8874-4a64-9137-a7fe056db11e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.552921 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.552907 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab9122ac-8874-4a64-9137-a7fe056db11e-istiod-ca-cert" pod "ab9122ac-8874-4a64-9137-a7fe056db11e" (UID: "ab9122ac-8874-4a64-9137-a7fe056db11e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:24.553176 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.553154 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baefa3c0-28be-428a-9591-f0ca1f07fafa-istiod-ca-cert" pod "baefa3c0-28be-428a-9591-f0ca1f07fafa" (UID: "baefa3c0-28be-428a-9591-f0ca1f07fafa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:24.553422 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.553402 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-data\") pod \"baefa3c0-28be-428a-9591-f0ca1f07fafa\" (UID: \"baefa3c0-28be-428a-9591-f0ca1f07fafa\") " Nov 04 01:34:24.554307 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.554281 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.554435 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.554313 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.554435 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.554329 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.554435 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.554342 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.554435 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.554359 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.554435 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.554374 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.554435 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.554387 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.554435 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.554404 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9122ac-8874-4a64-9137-a7fe056db11e-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.554435 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.554417 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baefa3c0-28be-428a-9591-f0ca1f07fafa-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.554846 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.554683 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-envoy" pod "baefa3c0-28be-428a-9591-f0ca1f07fafa" (UID: "baefa3c0-28be-428a-9591-f0ca1f07fafa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.554846 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.554709 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-data" pod "baefa3c0-28be-428a-9591-f0ca1f07fafa" (UID: "baefa3c0-28be-428a-9591-f0ca1f07fafa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.554846 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.554765 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-envoy" pod "ab9122ac-8874-4a64-9137-a7fe056db11e" (UID: "ab9122ac-8874-4a64-9137-a7fe056db11e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.555044 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.555022 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-token" pod "ab9122ac-8874-4a64-9137-a7fe056db11e" (UID: "ab9122ac-8874-4a64-9137-a7fe056db11e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:24.555429 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.555397 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-token" pod "baefa3c0-28be-428a-9591-f0ca1f07fafa" (UID: "baefa3c0-28be-428a-9591-f0ca1f07fafa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:24.556148 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.556100 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baefa3c0-28be-428a-9591-f0ca1f07fafa-kube-api-access-wkcjp" pod "baefa3c0-28be-428a-9591-f0ca1f07fafa" (UID: "baefa3c0-28be-428a-9591-f0ca1f07fafa"). InnerVolumeSpecName "kube-api-access-wkcjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:24.556257 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.556205 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab9122ac-8874-4a64-9137-a7fe056db11e-kube-api-access-79qks" pod "ab9122ac-8874-4a64-9137-a7fe056db11e" (UID: "ab9122ac-8874-4a64-9137-a7fe056db11e"). InnerVolumeSpecName "kube-api-access-79qks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:24.556403 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.556382 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-podinfo" pod "baefa3c0-28be-428a-9591-f0ca1f07fafa" (UID: "baefa3c0-28be-428a-9591-f0ca1f07fafa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:24.560815 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.560771 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-podinfo" pod "ab9122ac-8874-4a64-9137-a7fe056db11e" (UID: "ab9122ac-8874-4a64-9137-a7fe056db11e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:24.655572 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.655540 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.655733 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.655596 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.655733 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.655606 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.655733 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.655616 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkcjp\" (UniqueName: \"kubernetes.io/projected/baefa3c0-28be-428a-9591-f0ca1f07fafa-kube-api-access-wkcjp\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.655733 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.655625 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.655733 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.655632 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.655733 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.655641 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79qks\" (UniqueName: \"kubernetes.io/projected/ab9122ac-8874-4a64-9137-a7fe056db11e-kube-api-access-79qks\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.655733 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.655650 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9122ac-8874-4a64-9137-a7fe056db11e-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.655733 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.655659 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baefa3c0-28be-428a-9591-f0ca1f07fafa-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.719674 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.719644 4852 generic.go:356] "Generic (PLEG): container finished" podID="baefa3c0-28be-428a-9591-f0ca1f07fafa" containerID="b8bce3eaa168e4e58088218699c821309762b5813ac17399f4653bec114d0e74" exitCode=0 Nov 04 01:34:24.719875 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.719695 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" Nov 04 01:34:24.719875 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.719726 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" event={"ID":"baefa3c0-28be-428a-9591-f0ca1f07fafa","Type":"ContainerDied","Data":"b8bce3eaa168e4e58088218699c821309762b5813ac17399f4653bec114d0e74"} Nov 04 01:34:24.719875 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.719780 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p" event={"ID":"baefa3c0-28be-428a-9591-f0ca1f07fafa","Type":"ContainerDied","Data":"8fdd53a1e689ddedbdaf659d7f8018bf5054b8ce95641113e49102c8880f4f03"} Nov 04 01:34:24.719875 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.719810 4852 scope.go:122] "RemoveContainer" containerID="b8bce3eaa168e4e58088218699c821309762b5813ac17399f4653bec114d0e74" Nov 04 01:34:24.721139 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.721119 4852 generic.go:356] "Generic (PLEG): container finished" podID="ab9122ac-8874-4a64-9137-a7fe056db11e" containerID="341c1ef2ada34d68609175467758b55c188435bbdf83814e34ab1401e6866575" exitCode=0 Nov 04 01:34:24.721238 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.721157 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" Nov 04 01:34:24.721238 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.721194 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" event={"ID":"ab9122ac-8874-4a64-9137-a7fe056db11e","Type":"ContainerDied","Data":"341c1ef2ada34d68609175467758b55c188435bbdf83814e34ab1401e6866575"} Nov 04 01:34:24.721238 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.721248 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw" event={"ID":"ab9122ac-8874-4a64-9137-a7fe056db11e","Type":"ContainerDied","Data":"89930da40acbb8a0dd573c488ef91a926e1127d227773b8dc5929229d1b4386e"} Nov 04 01:34:24.731104 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.731089 4852 scope.go:122] "RemoveContainer" containerID="b8bce3eaa168e4e58088218699c821309762b5813ac17399f4653bec114d0e74" Nov 04 01:34:24.731457 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:34:24.731433 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8bce3eaa168e4e58088218699c821309762b5813ac17399f4653bec114d0e74\": container with ID starting with b8bce3eaa168e4e58088218699c821309762b5813ac17399f4653bec114d0e74 not found: ID does not exist" containerID="b8bce3eaa168e4e58088218699c821309762b5813ac17399f4653bec114d0e74" Nov 04 01:34:24.731534 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.731467 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8bce3eaa168e4e58088218699c821309762b5813ac17399f4653bec114d0e74"} err="failed to get container status \"b8bce3eaa168e4e58088218699c821309762b5813ac17399f4653bec114d0e74\": rpc error: code = NotFound desc = could not find container \"b8bce3eaa168e4e58088218699c821309762b5813ac17399f4653bec114d0e74\": container with ID starting with b8bce3eaa168e4e58088218699c821309762b5813ac17399f4653bec114d0e74 not found: ID does not exist" Nov 04 01:34:24.731534 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.731498 4852 scope.go:122] "RemoveContainer" containerID="341c1ef2ada34d68609175467758b55c188435bbdf83814e34ab1401e6866575" Nov 04 01:34:24.740596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.740554 4852 scope.go:122] "RemoveContainer" containerID="341c1ef2ada34d68609175467758b55c188435bbdf83814e34ab1401e6866575" Nov 04 01:34:24.740849 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:34:24.740831 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"341c1ef2ada34d68609175467758b55c188435bbdf83814e34ab1401e6866575\": container with ID starting with 341c1ef2ada34d68609175467758b55c188435bbdf83814e34ab1401e6866575 not found: ID does not exist" containerID="341c1ef2ada34d68609175467758b55c188435bbdf83814e34ab1401e6866575" Nov 04 01:34:24.740893 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.740857 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341c1ef2ada34d68609175467758b55c188435bbdf83814e34ab1401e6866575"} err="failed to get container status \"341c1ef2ada34d68609175467758b55c188435bbdf83814e34ab1401e6866575\": rpc error: code = NotFound desc = could not find container \"341c1ef2ada34d68609175467758b55c188435bbdf83814e34ab1401e6866575\": container with ID starting with 341c1ef2ada34d68609175467758b55c188435bbdf83814e34ab1401e6866575 not found: ID does not exist" Nov 04 01:34:24.786080 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.784042 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw"] Nov 04 01:34:24.788579 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.788531 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-p4gpw"] Nov 04 01:34:24.806853 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.806832 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p"] Nov 04 01:34:24.811774 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.811757 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-m647p"] Nov 04 01:34:24.906229 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.906197 4852 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 04 01:34:24.906392 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:24.906261 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" podUID="1efe83d4-74d6-431e-91fd-314ce0f5d116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 04 01:34:25.044587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:25.040974 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:25.778214 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:25.778183 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:25.910321 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:25.910295 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:34:25.911531 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:25.911510 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:34:25.938533 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:25.938486 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:25.998806 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:25.998767 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab9122ac-8874-4a64-9137-a7fe056db11e" path="/var/lib/kubelet/pods/ab9122ac-8874-4a64-9137-a7fe056db11e/volumes" Nov 04 01:34:25.999703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:25.999677 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baefa3c0-28be-428a-9591-f0ca1f07fafa" path="/var/lib/kubelet/pods/baefa3c0-28be-428a-9591-f0ca1f07fafa/volumes" Nov 04 01:34:26.257651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:26.256981 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:26.511167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:26.511068 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw"] Nov 04 01:34:26.527332 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:26.527289 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6" podUID="456612ef-2ef3-4b16-bab3-1277c84311c5" containerName="payload-processing" containerID="cri-o://83940cffbec9e8adf33b391cef72116ec984dc3d29e1a93574d6bbe69d11595b" gracePeriod=25 Nov 04 01:34:26.662588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:26.644094 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:26.773724 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:26.773626 4852 generic.go:356] "Generic (PLEG): container finished" podID="456612ef-2ef3-4b16-bab3-1277c84311c5" containerID="83940cffbec9e8adf33b391cef72116ec984dc3d29e1a93574d6bbe69d11595b" exitCode=0 Nov 04 01:34:26.774305 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:26.774274 4852 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" secret="" err="secret \"e2e-shared-a-w3-2fff32-openshift-default-dockercfg-x2msp\" not found" Nov 04 01:34:26.774952 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:26.774920 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6" event={"ID":"456612ef-2ef3-4b16-bab3-1277c84311c5","Type":"ContainerDied","Data":"83940cffbec9e8adf33b391cef72116ec984dc3d29e1a93574d6bbe69d11595b"} Nov 04 01:34:26.962723 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:26.962693 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6" Nov 04 01:34:27.026699 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:27.026628 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/456612ef-2ef3-4b16-bab3-1277c84311c5-kube-api-access-ngqmt\" (UniqueName: \"kubernetes.io/projected/456612ef-2ef3-4b16-bab3-1277c84311c5-kube-api-access-ngqmt\") pod \"456612ef-2ef3-4b16-bab3-1277c84311c5\" (UID: \"456612ef-2ef3-4b16-bab3-1277c84311c5\") " Nov 04 01:34:27.026849 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:27.026723 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/456612ef-2ef3-4b16-bab3-1277c84311c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/456612ef-2ef3-4b16-bab3-1277c84311c5-plugins-config-volume\") pod \"456612ef-2ef3-4b16-bab3-1277c84311c5\" (UID: \"456612ef-2ef3-4b16-bab3-1277c84311c5\") " Nov 04 01:34:27.027130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:27.027108 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/456612ef-2ef3-4b16-bab3-1277c84311c5-plugins-config-volume" pod "456612ef-2ef3-4b16-bab3-1277c84311c5" (UID: "456612ef-2ef3-4b16-bab3-1277c84311c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:27.027276 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:27.027261 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/456612ef-2ef3-4b16-bab3-1277c84311c5-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:27.028848 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:27.028826 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/456612ef-2ef3-4b16-bab3-1277c84311c5-kube-api-access-ngqmt" pod "456612ef-2ef3-4b16-bab3-1277c84311c5" (UID: "456612ef-2ef3-4b16-bab3-1277c84311c5"). InnerVolumeSpecName "kube-api-access-ngqmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:27.094198 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:27.093317 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:27.133587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:27.132741 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngqmt\" (UniqueName: \"kubernetes.io/projected/456612ef-2ef3-4b16-bab3-1277c84311c5-kube-api-access-ngqmt\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:27.431269 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:27.431181 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:27.782640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:27.782614 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6" Nov 04 01:34:27.782799 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:27.782641 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6" event={"ID":"456612ef-2ef3-4b16-bab3-1277c84311c5","Type":"ContainerDied","Data":"64f8cb80a5761a9ea8287603f47ddd233cc69ebd86c36478cb6f7890f52d803e"} Nov 04 01:34:27.782799 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:27.782706 4852 scope.go:122] "RemoveContainer" containerID="83940cffbec9e8adf33b391cef72116ec984dc3d29e1a93574d6bbe69d11595b" Nov 04 01:34:27.782906 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:27.782816 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" podUID="1efe83d4-74d6-431e-91fd-314ce0f5d116" containerName="istio-proxy" containerID="cri-o://e5c328e2adcecb5bdf01a8f879927b213e733e73e6ec04e0b543fba2c2491d28" gracePeriod=30 Nov 04 01:34:27.797134 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:27.797096 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:27.840396 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:27.840367 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6"] Nov 04 01:34:27.845939 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:27.845917 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-586541-55c5b68d94-pmxj6"] Nov 04 01:34:28.009318 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:28.009273 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="456612ef-2ef3-4b16-bab3-1277c84311c5" path="/var/lib/kubelet/pods/456612ef-2ef3-4b16-bab3-1277c84311c5/volumes" Nov 04 01:34:28.024478 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:28.024350 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:28.198165 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:28.198081 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:28.752588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:28.743747 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:29.629589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:29.613260 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:29.866383 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:34:29.866348 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:34:30.060867 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.060836 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6"] Nov 04 01:34:30.061439 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.061405 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" podUID="9961d5eb-fd3c-4651-96a8-861577e1ad21" containerName="maas-api" containerID="cri-o://e366d7e972425458663f934dfc1bfa8bbca855ddd3e2561cd0f222a78ea8fe36" gracePeriod=30 Nov 04 01:34:30.153745 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.153715 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:30.511000 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.509183 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:30.519586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.512112 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" containerID="cri-o://6e4774dbed44e28893d7cc9d66e85863f7b972b92ba739a6336a66d7454d9809" gracePeriod=45 Nov 04 01:34:30.623799 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.623071 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": dial tcp 10.134.0.39:8000: connect: connection refused" Nov 04 01:34:30.643708 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.643673 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:30.687013 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.686983 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" Nov 04 01:34:30.750211 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.750178 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9961d5eb-fd3c-4651-96a8-861577e1ad21-kube-api-access-xhwwl\" (UniqueName: \"kubernetes.io/projected/9961d5eb-fd3c-4651-96a8-861577e1ad21-kube-api-access-xhwwl\") pod \"9961d5eb-fd3c-4651-96a8-861577e1ad21\" (UID: \"9961d5eb-fd3c-4651-96a8-861577e1ad21\") " Nov 04 01:34:30.750394 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.750332 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9961d5eb-fd3c-4651-96a8-861577e1ad21-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9961d5eb-fd3c-4651-96a8-861577e1ad21-metrics-certs\") pod \"9961d5eb-fd3c-4651-96a8-861577e1ad21\" (UID: \"9961d5eb-fd3c-4651-96a8-861577e1ad21\") " Nov 04 01:34:30.750394 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.750378 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9961d5eb-fd3c-4651-96a8-861577e1ad21-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9961d5eb-fd3c-4651-96a8-861577e1ad21-maas-api-tls\") pod \"9961d5eb-fd3c-4651-96a8-861577e1ad21\" (UID: \"9961d5eb-fd3c-4651-96a8-861577e1ad21\") " Nov 04 01:34:30.750534 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.750418 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9961d5eb-fd3c-4651-96a8-861577e1ad21-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9961d5eb-fd3c-4651-96a8-861577e1ad21-openshift-service-ca\") pod \"9961d5eb-fd3c-4651-96a8-861577e1ad21\" (UID: \"9961d5eb-fd3c-4651-96a8-861577e1ad21\") " Nov 04 01:34:30.753270 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.751473 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9961d5eb-fd3c-4651-96a8-861577e1ad21-openshift-service-ca" pod "9961d5eb-fd3c-4651-96a8-861577e1ad21" (UID: "9961d5eb-fd3c-4651-96a8-861577e1ad21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:30.758262 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.758076 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9961d5eb-fd3c-4651-96a8-861577e1ad21-maas-api-tls" pod "9961d5eb-fd3c-4651-96a8-861577e1ad21" (UID: "9961d5eb-fd3c-4651-96a8-861577e1ad21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:30.758262 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.758131 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9961d5eb-fd3c-4651-96a8-861577e1ad21-metrics-certs" pod "9961d5eb-fd3c-4651-96a8-861577e1ad21" (UID: "9961d5eb-fd3c-4651-96a8-861577e1ad21"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:30.758262 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.758183 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9961d5eb-fd3c-4651-96a8-861577e1ad21-kube-api-access-xhwwl" pod "9961d5eb-fd3c-4651-96a8-861577e1ad21" (UID: "9961d5eb-fd3c-4651-96a8-861577e1ad21"). InnerVolumeSpecName "kube-api-access-xhwwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:30.798144 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.798100 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:30.807345 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.807317 4852 generic.go:356] "Generic (PLEG): container finished" podID="9961d5eb-fd3c-4651-96a8-861577e1ad21" containerID="e366d7e972425458663f934dfc1bfa8bbca855ddd3e2561cd0f222a78ea8fe36" exitCode=0 Nov 04 01:34:30.807482 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.807380 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" Nov 04 01:34:30.807482 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.807407 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" event={"ID":"9961d5eb-fd3c-4651-96a8-861577e1ad21","Type":"ContainerDied","Data":"e366d7e972425458663f934dfc1bfa8bbca855ddd3e2561cd0f222a78ea8fe36"} Nov 04 01:34:30.808472 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.808449 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6" event={"ID":"9961d5eb-fd3c-4651-96a8-861577e1ad21","Type":"ContainerDied","Data":"7843e0d21f30e3b8590cc65a9eea52c4588ba4445699901cf4b5743897793071"} Nov 04 01:34:30.808624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.808475 4852 scope.go:122] "RemoveContainer" containerID="e366d7e972425458663f934dfc1bfa8bbca855ddd3e2561cd0f222a78ea8fe36" Nov 04 01:34:30.809273 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.809245 4852 generic.go:356] "Generic (PLEG): container finished" podID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerID="6e4774dbed44e28893d7cc9d66e85863f7b972b92ba739a6336a66d7454d9809" exitCode=0 Nov 04 01:34:30.809386 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.809288 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" event={"ID":"761be4d8-0736-4a22-bd18-2729dbc85a11","Type":"ContainerDied","Data":"6e4774dbed44e28893d7cc9d66e85863f7b972b92ba739a6336a66d7454d9809"} Nov 04 01:34:30.821667 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.821646 4852 scope.go:122] "RemoveContainer" containerID="e366d7e972425458663f934dfc1bfa8bbca855ddd3e2561cd0f222a78ea8fe36" Nov 04 01:34:30.821783 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.821682 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:34:30.821943 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:34:30.821907 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e366d7e972425458663f934dfc1bfa8bbca855ddd3e2561cd0f222a78ea8fe36\": container with ID starting with e366d7e972425458663f934dfc1bfa8bbca855ddd3e2561cd0f222a78ea8fe36 not found: ID does not exist" containerID="e366d7e972425458663f934dfc1bfa8bbca855ddd3e2561cd0f222a78ea8fe36" Nov 04 01:34:30.822030 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.821938 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e366d7e972425458663f934dfc1bfa8bbca855ddd3e2561cd0f222a78ea8fe36"} err="failed to get container status \"e366d7e972425458663f934dfc1bfa8bbca855ddd3e2561cd0f222a78ea8fe36\": rpc error: code = NotFound desc = could not find container \"e366d7e972425458663f934dfc1bfa8bbca855ddd3e2561cd0f222a78ea8fe36\": container with ID starting with e366d7e972425458663f934dfc1bfa8bbca855ddd3e2561cd0f222a78ea8fe36 not found: ID does not exist" Nov 04 01:34:30.840347 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.840314 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6"] Nov 04 01:34:30.843996 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.843959 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9c8bf5-7976d44897-xrrp6"] Nov 04 01:34:30.852128 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.852106 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-model-cache\") pod \"761be4d8-0736-4a22-bd18-2729dbc85a11\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " Nov 04 01:34:30.852265 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.852205 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/761be4d8-0736-4a22-bd18-2729dbc85a11-kube-api-access-p66jd\" (UniqueName: \"kubernetes.io/projected/761be4d8-0736-4a22-bd18-2729dbc85a11-kube-api-access-p66jd\") pod \"761be4d8-0736-4a22-bd18-2729dbc85a11\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " Nov 04 01:34:30.852265 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.852239 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/761be4d8-0736-4a22-bd18-2729dbc85a11-tls-certs\" (UniqueName: \"kubernetes.io/secret/761be4d8-0736-4a22-bd18-2729dbc85a11-tls-certs\") pod \"761be4d8-0736-4a22-bd18-2729dbc85a11\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " Nov 04 01:34:30.852265 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.852261 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-dshm\" (UniqueName: \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-dshm\") pod \"761be4d8-0736-4a22-bd18-2729dbc85a11\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " Nov 04 01:34:30.852482 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.852291 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-home\" (UniqueName: \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-home\") pod \"761be4d8-0736-4a22-bd18-2729dbc85a11\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " Nov 04 01:34:30.852482 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.852359 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-tmp-dir\") pod \"761be4d8-0736-4a22-bd18-2729dbc85a11\" (UID: \"761be4d8-0736-4a22-bd18-2729dbc85a11\") " Nov 04 01:34:30.852482 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.852370 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-model-cache" pod "761be4d8-0736-4a22-bd18-2729dbc85a11" (UID: "761be4d8-0736-4a22-bd18-2729dbc85a11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:30.852674 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.852633 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-home" pod "761be4d8-0736-4a22-bd18-2729dbc85a11" (UID: "761be4d8-0736-4a22-bd18-2729dbc85a11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:30.852674 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.852659 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-tmp-dir" pod "761be4d8-0736-4a22-bd18-2729dbc85a11" (UID: "761be4d8-0736-4a22-bd18-2729dbc85a11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:30.853231 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.853209 4852 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-home\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:30.853294 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.853239 4852 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-tmp-dir\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:30.853294 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.853256 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhwwl\" (UniqueName: \"kubernetes.io/projected/9961d5eb-fd3c-4651-96a8-861577e1ad21-kube-api-access-xhwwl\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:30.853294 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.853271 4852 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9961d5eb-fd3c-4651-96a8-861577e1ad21-metrics-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:30.853294 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.853285 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9961d5eb-fd3c-4651-96a8-861577e1ad21-maas-api-tls\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:30.853422 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.853298 4852 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-model-cache\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:30.853422 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.853311 4852 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9961d5eb-fd3c-4651-96a8-861577e1ad21-openshift-service-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:30.854712 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.854687 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-dshm" pod "761be4d8-0736-4a22-bd18-2729dbc85a11" (UID: "761be4d8-0736-4a22-bd18-2729dbc85a11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:30.854987 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.854971 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/761be4d8-0736-4a22-bd18-2729dbc85a11-tls-certs" pod "761be4d8-0736-4a22-bd18-2729dbc85a11" (UID: "761be4d8-0736-4a22-bd18-2729dbc85a11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:30.855060 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.855017 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/761be4d8-0736-4a22-bd18-2729dbc85a11-kube-api-access-p66jd" pod "761be4d8-0736-4a22-bd18-2729dbc85a11" (UID: "761be4d8-0736-4a22-bd18-2729dbc85a11"). InnerVolumeSpecName "kube-api-access-p66jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:30.954619 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.954534 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p66jd\" (UniqueName: \"kubernetes.io/projected/761be4d8-0736-4a22-bd18-2729dbc85a11-kube-api-access-p66jd\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:30.954619 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.954595 4852 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761be4d8-0736-4a22-bd18-2729dbc85a11-tls-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:30.954619 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:30.954607 4852 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/761be4d8-0736-4a22-bd18-2729dbc85a11-dshm\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:31.277242 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:31.277133 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:31.813035 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:31.813008 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:31.817514 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:31.817489 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" Nov 04 01:34:31.817646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:31.817531 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq" event={"ID":"761be4d8-0736-4a22-bd18-2729dbc85a11","Type":"ContainerDied","Data":"254c73d15ae1233748a9a67ac7fa371a804bb3bf994e42f616a602d2ab4f4bbf"} Nov 04 01:34:31.817646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:31.817609 4852 scope.go:122] "RemoveContainer" containerID="6e4774dbed44e28893d7cc9d66e85863f7b972b92ba739a6336a66d7454d9809" Nov 04 01:34:31.841448 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:31.841425 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq"] Nov 04 01:34:31.843949 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:31.843928 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-586541/premium-w1-586541-kserve-6df644b944-987zq"] Nov 04 01:34:31.997959 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:31.997925 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" path="/var/lib/kubelet/pods/761be4d8-0736-4a22-bd18-2729dbc85a11/volumes" Nov 04 01:34:31.998646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:31.998599 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9961d5eb-fd3c-4651-96a8-861577e1ad21" path="/var/lib/kubelet/pods/9961d5eb-fd3c-4651-96a8-861577e1ad21/volumes" Nov 04 01:34:31.999933 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:31.999907 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:32.186945 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:32.185912 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:32.866593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:32.865420 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:33.086856 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.086831 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:34:33.204310 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.204284 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1efe83d4-74d6-431e-91fd-314ce0f5d116-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1efe83d4-74d6-431e-91fd-314ce0f5d116-istiod-ca-cert\") pod \"1efe83d4-74d6-431e-91fd-314ce0f5d116\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " Nov 04 01:34:33.204468 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.204320 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-credential-socket\") pod \"1efe83d4-74d6-431e-91fd-314ce0f5d116\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " Nov 04 01:34:33.204468 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.204353 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-token\" (UniqueName: \"kubernetes.io/projected/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-token\") pod \"1efe83d4-74d6-431e-91fd-314ce0f5d116\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " Nov 04 01:34:33.204468 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.204383 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1efe83d4-74d6-431e-91fd-314ce0f5d116-kube-api-access-9zvk9\" (UniqueName: \"kubernetes.io/projected/1efe83d4-74d6-431e-91fd-314ce0f5d116-kube-api-access-9zvk9\") pod \"1efe83d4-74d6-431e-91fd-314ce0f5d116\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " Nov 04 01:34:33.204468 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.204418 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-workload-socket\") pod \"1efe83d4-74d6-431e-91fd-314ce0f5d116\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " Nov 04 01:34:33.204468 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.204462 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-envoy\") pod \"1efe83d4-74d6-431e-91fd-314ce0f5d116\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " Nov 04 01:34:33.204755 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.204507 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-workload-certs\") pod \"1efe83d4-74d6-431e-91fd-314ce0f5d116\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " Nov 04 01:34:33.204755 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.204618 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-credential-socket" pod "1efe83d4-74d6-431e-91fd-314ce0f5d116" (UID: "1efe83d4-74d6-431e-91fd-314ce0f5d116"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:33.204755 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.204676 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-data\") pod \"1efe83d4-74d6-431e-91fd-314ce0f5d116\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " Nov 04 01:34:33.204755 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.204724 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-podinfo\") pod \"1efe83d4-74d6-431e-91fd-314ce0f5d116\" (UID: \"1efe83d4-74d6-431e-91fd-314ce0f5d116\") " Nov 04 01:34:33.204963 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.204781 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-workload-socket" pod "1efe83d4-74d6-431e-91fd-314ce0f5d116" (UID: "1efe83d4-74d6-431e-91fd-314ce0f5d116"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:33.204963 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.204885 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-workload-certs" pod "1efe83d4-74d6-431e-91fd-314ce0f5d116" (UID: "1efe83d4-74d6-431e-91fd-314ce0f5d116"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:33.204963 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.204915 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-data" pod "1efe83d4-74d6-431e-91fd-314ce0f5d116" (UID: "1efe83d4-74d6-431e-91fd-314ce0f5d116"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:33.205128 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.204787 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1efe83d4-74d6-431e-91fd-314ce0f5d116-istiod-ca-cert" pod "1efe83d4-74d6-431e-91fd-314ce0f5d116" (UID: "1efe83d4-74d6-431e-91fd-314ce0f5d116"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:33.205486 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.205463 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1efe83d4-74d6-431e-91fd-314ce0f5d116-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:33.205588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.205491 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:33.205588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.205508 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:33.205588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.205521 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:33.205588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.205539 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:33.206780 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.206758 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-envoy" pod "1efe83d4-74d6-431e-91fd-314ce0f5d116" (UID: "1efe83d4-74d6-431e-91fd-314ce0f5d116"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:33.207049 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.207030 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-token" pod "1efe83d4-74d6-431e-91fd-314ce0f5d116" (UID: "1efe83d4-74d6-431e-91fd-314ce0f5d116"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:33.207104 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.207062 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1efe83d4-74d6-431e-91fd-314ce0f5d116-kube-api-access-9zvk9" pod "1efe83d4-74d6-431e-91fd-314ce0f5d116" (UID: "1efe83d4-74d6-431e-91fd-314ce0f5d116"). InnerVolumeSpecName "kube-api-access-9zvk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:33.207104 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.207064 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-podinfo" pod "1efe83d4-74d6-431e-91fd-314ce0f5d116" (UID: "1efe83d4-74d6-431e-91fd-314ce0f5d116"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:33.307217 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.307183 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:33.307217 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.307213 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zvk9\" (UniqueName: \"kubernetes.io/projected/1efe83d4-74d6-431e-91fd-314ce0f5d116-kube-api-access-9zvk9\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:33.307217 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.307223 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:33.307438 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.307232 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1efe83d4-74d6-431e-91fd-314ce0f5d116-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:33.447472 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.447389 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:33.676980 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.676942 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l"] Nov 04 01:34:33.677248 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.677219 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" podUID="d26cecde-b16d-4fe1-add0-41a6d0a5301a" containerName="istio-proxy" containerID="cri-o://08d2bc451cb6986a1954ebf9622dbb315cb8fccc8772f554e26c0f33ee500a44" gracePeriod=30 Nov 04 01:34:33.694330 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.694306 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr"] Nov 04 01:34:33.694546 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.694525 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" podUID="7619e54f-752b-428f-b955-b2dec4471c5b" containerName="istio-proxy" containerID="cri-o://7b84e0ed4b23e5e32b42ee9aa15b1acf044ce593c365273c5472ff267e6dc873" gracePeriod=30 Nov 04 01:34:33.792534 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.792472 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:33.835636 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.835600 4852 generic.go:356] "Generic (PLEG): container finished" podID="1efe83d4-74d6-431e-91fd-314ce0f5d116" containerID="e5c328e2adcecb5bdf01a8f879927b213e733e73e6ec04e0b543fba2c2491d28" exitCode=0 Nov 04 01:34:33.835809 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.835665 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" event={"ID":"1efe83d4-74d6-431e-91fd-314ce0f5d116","Type":"ContainerDied","Data":"e5c328e2adcecb5bdf01a8f879927b213e733e73e6ec04e0b543fba2c2491d28"} Nov 04 01:34:33.835809 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.835672 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" Nov 04 01:34:33.835809 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.835713 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw" event={"ID":"1efe83d4-74d6-431e-91fd-314ce0f5d116","Type":"ContainerDied","Data":"3900b6741ad20b15c0389824907b063dc6205ff734e79fc863965dbee899b3ac"} Nov 04 01:34:33.835809 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.835734 4852 scope.go:122] "RemoveContainer" containerID="e5c328e2adcecb5bdf01a8f879927b213e733e73e6ec04e0b543fba2c2491d28" Nov 04 01:34:33.851720 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.850132 4852 scope.go:122] "RemoveContainer" containerID="392fea8c004cca56e05fd1f56f459bc5f07ec94d8d4b6daf9bb2fb2fb7596c91" Nov 04 01:34:33.862662 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.862640 4852 scope.go:122] "RemoveContainer" containerID="e5c328e2adcecb5bdf01a8f879927b213e733e73e6ec04e0b543fba2c2491d28" Nov 04 01:34:33.880588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.878548 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw"] Nov 04 01:34:33.882005 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:34:33.881971 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5c328e2adcecb5bdf01a8f879927b213e733e73e6ec04e0b543fba2c2491d28\": container with ID starting with e5c328e2adcecb5bdf01a8f879927b213e733e73e6ec04e0b543fba2c2491d28 not found: ID does not exist" containerID="e5c328e2adcecb5bdf01a8f879927b213e733e73e6ec04e0b543fba2c2491d28" Nov 04 01:34:33.882233 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.882204 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5c328e2adcecb5bdf01a8f879927b213e733e73e6ec04e0b543fba2c2491d28"} err="failed to get container status \"e5c328e2adcecb5bdf01a8f879927b213e733e73e6ec04e0b543fba2c2491d28\": rpc error: code = NotFound desc = could not find container \"e5c328e2adcecb5bdf01a8f879927b213e733e73e6ec04e0b543fba2c2491d28\": container with ID starting with e5c328e2adcecb5bdf01a8f879927b213e733e73e6ec04e0b543fba2c2491d28 not found: ID does not exist" Nov 04 01:34:33.882372 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.882357 4852 scope.go:122] "RemoveContainer" containerID="392fea8c004cca56e05fd1f56f459bc5f07ec94d8d4b6daf9bb2fb2fb7596c91" Nov 04 01:34:33.883086 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:34:33.883019 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"392fea8c004cca56e05fd1f56f459bc5f07ec94d8d4b6daf9bb2fb2fb7596c91\": container with ID starting with 392fea8c004cca56e05fd1f56f459bc5f07ec94d8d4b6daf9bb2fb2fb7596c91 not found: ID does not exist" containerID="392fea8c004cca56e05fd1f56f459bc5f07ec94d8d4b6daf9bb2fb2fb7596c91" Nov 04 01:34:33.883086 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.883056 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"392fea8c004cca56e05fd1f56f459bc5f07ec94d8d4b6daf9bb2fb2fb7596c91"} err="failed to get container status \"392fea8c004cca56e05fd1f56f459bc5f07ec94d8d4b6daf9bb2fb2fb7596c91\": rpc error: code = NotFound desc = could not find container \"392fea8c004cca56e05fd1f56f459bc5f07ec94d8d4b6daf9bb2fb2fb7596c91\": container with ID starting with 392fea8c004cca56e05fd1f56f459bc5f07ec94d8d4b6daf9bb2fb2fb7596c91 not found: ID does not exist" Nov 04 01:34:33.891131 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:33.889073 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2fff32-openshift-default-67975d4f6c-92rgw"] Nov 04 01:34:34.006587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:34.001794 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:34.034602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:34.034419 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1efe83d4-74d6-431e-91fd-314ce0f5d116" path="/var/lib/kubelet/pods/1efe83d4-74d6-431e-91fd-314ce0f5d116/volumes" Nov 04 01:34:34.950732 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:34.943190 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:35.216620 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:35.216545 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:35.797489 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:35.797457 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:35.909771 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:35.909741 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc"] Nov 04 01:34:35.956853 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:35.956823 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:36.085773 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:36.085697 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:34:36.085773 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:36.085754 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" podUID="7619e54f-752b-428f-b955-b2dec4471c5b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:34:36.097133 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:36.097107 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:36.281587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:36.277321 4852 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:34:36.281587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:36.277378 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" podUID="d26cecde-b16d-4fe1-add0-41a6d0a5301a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:34:36.508484 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:36.508454 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:36.782626 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:36.778552 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:36.944734 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:36.944705 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:37.281540 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:37.281509 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:37.973087 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:37.973054 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m"] Nov 04 01:34:38.002217 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:38.002186 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb"] Nov 04 01:34:38.108228 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:38.108196 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:38.280589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:38.275180 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:38.870996 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:38.870955 4852 generic.go:356] "Generic (PLEG): container finished" podID="7619e54f-752b-428f-b955-b2dec4471c5b" containerID="7b84e0ed4b23e5e32b42ee9aa15b1acf044ce593c365273c5472ff267e6dc873" exitCode=0 Nov 04 01:34:38.871895 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:38.871129 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" event={"ID":"7619e54f-752b-428f-b955-b2dec4471c5b","Type":"ContainerDied","Data":"7b84e0ed4b23e5e32b42ee9aa15b1acf044ce593c365273c5472ff267e6dc873"} Nov 04 01:34:38.873278 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:38.872950 4852 generic.go:356] "Generic (PLEG): container finished" podID="d26cecde-b16d-4fe1-add0-41a6d0a5301a" containerID="08d2bc451cb6986a1954ebf9622dbb315cb8fccc8772f554e26c0f33ee500a44" exitCode=0 Nov 04 01:34:38.873278 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:38.873001 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" event={"ID":"d26cecde-b16d-4fe1-add0-41a6d0a5301a","Type":"ContainerDied","Data":"08d2bc451cb6986a1954ebf9622dbb315cb8fccc8772f554e26c0f33ee500a44"} Nov 04 01:34:38.969452 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:38.969431 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:34:38.977502 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:38.977474 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:39.001033 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.001011 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:34:39.036343 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.036314 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-workload-certs\") pod \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " Nov 04 01:34:39.036466 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.036395 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26cecde-b16d-4fe1-add0-41a6d0a5301a-kube-api-access-xs2l6\" (UniqueName: \"kubernetes.io/projected/d26cecde-b16d-4fe1-add0-41a6d0a5301a-kube-api-access-xs2l6\") pod \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " Nov 04 01:34:39.036540 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.036473 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-podinfo\") pod \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " Nov 04 01:34:39.036617 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.036541 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-data\") pod \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " Nov 04 01:34:39.036686 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.036607 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-workload-certs" pod "d26cecde-b16d-4fe1-add0-41a6d0a5301a" (UID: "d26cecde-b16d-4fe1-add0-41a6d0a5301a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.036686 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.036656 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-envoy\") pod \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " Nov 04 01:34:39.036798 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.036686 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-workload-socket\") pod \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " Nov 04 01:34:39.036798 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.036759 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-token\" (UniqueName: \"kubernetes.io/projected/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-token\") pod \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " Nov 04 01:34:39.036798 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.036794 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-credential-socket\") pod \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " Nov 04 01:34:39.036954 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.036858 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istiod-ca-cert\") pod \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\" (UID: \"d26cecde-b16d-4fe1-add0-41a6d0a5301a\") " Nov 04 01:34:39.037416 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.037387 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-credential-socket" pod "d26cecde-b16d-4fe1-add0-41a6d0a5301a" (UID: "d26cecde-b16d-4fe1-add0-41a6d0a5301a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.037587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.037437 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.037587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.037498 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-workload-socket" pod "d26cecde-b16d-4fe1-add0-41a6d0a5301a" (UID: "d26cecde-b16d-4fe1-add0-41a6d0a5301a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.037735 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.037665 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istiod-ca-cert" pod "d26cecde-b16d-4fe1-add0-41a6d0a5301a" (UID: "d26cecde-b16d-4fe1-add0-41a6d0a5301a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:39.039480 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.039451 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-envoy" pod "d26cecde-b16d-4fe1-add0-41a6d0a5301a" (UID: "d26cecde-b16d-4fe1-add0-41a6d0a5301a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.039644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.039603 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-data" pod "d26cecde-b16d-4fe1-add0-41a6d0a5301a" (UID: "d26cecde-b16d-4fe1-add0-41a6d0a5301a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.040072 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.040049 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-token" pod "d26cecde-b16d-4fe1-add0-41a6d0a5301a" (UID: "d26cecde-b16d-4fe1-add0-41a6d0a5301a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:39.040322 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.040298 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-podinfo" pod "d26cecde-b16d-4fe1-add0-41a6d0a5301a" (UID: "d26cecde-b16d-4fe1-add0-41a6d0a5301a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:39.041856 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.041830 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26cecde-b16d-4fe1-add0-41a6d0a5301a-kube-api-access-xs2l6" pod "d26cecde-b16d-4fe1-add0-41a6d0a5301a" (UID: "d26cecde-b16d-4fe1-add0-41a6d0a5301a"). InnerVolumeSpecName "kube-api-access-xs2l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:39.138798 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.138769 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-credential-socket\") pod \"7619e54f-752b-428f-b955-b2dec4471c5b\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " Nov 04 01:34:39.138954 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.138893 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-workload-certs\") pod \"7619e54f-752b-428f-b955-b2dec4471c5b\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " Nov 04 01:34:39.138954 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.138917 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-istio-envoy\") pod \"7619e54f-752b-428f-b955-b2dec4471c5b\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " Nov 04 01:34:39.138954 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.138954 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7619e54f-752b-428f-b955-b2dec4471c5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7619e54f-752b-428f-b955-b2dec4471c5b-istiod-ca-cert\") pod \"7619e54f-752b-428f-b955-b2dec4471c5b\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " Nov 04 01:34:39.139130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.138999 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7619e54f-752b-428f-b955-b2dec4471c5b-kube-api-access-965lz\" (UniqueName: \"kubernetes.io/projected/7619e54f-752b-428f-b955-b2dec4471c5b-kube-api-access-965lz\") pod \"7619e54f-752b-428f-b955-b2dec4471c5b\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " Nov 04 01:34:39.139130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.139057 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-workload-socket\") pod \"7619e54f-752b-428f-b955-b2dec4471c5b\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " Nov 04 01:34:39.139130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.139074 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-credential-socket" pod "7619e54f-752b-428f-b955-b2dec4471c5b" (UID: "7619e54f-752b-428f-b955-b2dec4471c5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.139293 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.139140 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7619e54f-752b-428f-b955-b2dec4471c5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7619e54f-752b-428f-b955-b2dec4471c5b-istio-podinfo\") pod \"7619e54f-752b-428f-b955-b2dec4471c5b\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " Nov 04 01:34:39.139293 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.139195 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-istio-data\") pod \"7619e54f-752b-428f-b955-b2dec4471c5b\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " Nov 04 01:34:39.139293 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.139224 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7619e54f-752b-428f-b955-b2dec4471c5b-istio-token\" (UniqueName: \"kubernetes.io/projected/7619e54f-752b-428f-b955-b2dec4471c5b-istio-token\") pod \"7619e54f-752b-428f-b955-b2dec4471c5b\" (UID: \"7619e54f-752b-428f-b955-b2dec4471c5b\") " Nov 04 01:34:39.139293 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.139222 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-workload-certs" pod "7619e54f-752b-428f-b955-b2dec4471c5b" (UID: "7619e54f-752b-428f-b955-b2dec4471c5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.139502 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.139361 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-workload-socket" pod "7619e54f-752b-428f-b955-b2dec4471c5b" (UID: "7619e54f-752b-428f-b955-b2dec4471c5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.139502 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.139392 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7619e54f-752b-428f-b955-b2dec4471c5b-istiod-ca-cert" pod "7619e54f-752b-428f-b955-b2dec4471c5b" (UID: "7619e54f-752b-428f-b955-b2dec4471c5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:39.139649 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.139633 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-istio-data" pod "7619e54f-752b-428f-b955-b2dec4471c5b" (UID: "7619e54f-752b-428f-b955-b2dec4471c5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.139964 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.139906 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.140237 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.140220 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.140313 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.140243 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.140313 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.140258 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7619e54f-752b-428f-b955-b2dec4471c5b-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.140313 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.140272 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.140313 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.140285 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.140313 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.140299 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.140313 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.140313 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.140685 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.140326 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.140685 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.140339 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.140685 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.140353 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs2l6\" (UniqueName: \"kubernetes.io/projected/d26cecde-b16d-4fe1-add0-41a6d0a5301a-kube-api-access-xs2l6\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.140685 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.140368 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.140685 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.140383 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26cecde-b16d-4fe1-add0-41a6d0a5301a-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.141796 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.141773 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-istio-envoy" pod "7619e54f-752b-428f-b955-b2dec4471c5b" (UID: "7619e54f-752b-428f-b955-b2dec4471c5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.141995 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.141974 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7619e54f-752b-428f-b955-b2dec4471c5b-istio-podinfo" pod "7619e54f-752b-428f-b955-b2dec4471c5b" (UID: "7619e54f-752b-428f-b955-b2dec4471c5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:39.142624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.142597 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7619e54f-752b-428f-b955-b2dec4471c5b-kube-api-access-965lz" pod "7619e54f-752b-428f-b955-b2dec4471c5b" (UID: "7619e54f-752b-428f-b955-b2dec4471c5b"). InnerVolumeSpecName "kube-api-access-965lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:39.142835 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.142819 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7619e54f-752b-428f-b955-b2dec4471c5b-istio-token" pod "7619e54f-752b-428f-b955-b2dec4471c5b" (UID: "7619e54f-752b-428f-b955-b2dec4471c5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:39.185366 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.185336 4852 scope.go:122] "RemoveContainer" containerID="08d2bc451cb6986a1954ebf9622dbb315cb8fccc8772f554e26c0f33ee500a44" Nov 04 01:34:39.195842 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.195821 4852 scope.go:122] "RemoveContainer" containerID="7b84e0ed4b23e5e32b42ee9aa15b1acf044ce593c365273c5472ff267e6dc873" Nov 04 01:34:39.254606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.253051 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7619e54f-752b-428f-b955-b2dec4471c5b-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.254606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.253086 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7619e54f-752b-428f-b955-b2dec4471c5b-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.254606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.253105 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7619e54f-752b-428f-b955-b2dec4471c5b-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.254606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.253120 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-965lz\" (UniqueName: \"kubernetes.io/projected/7619e54f-752b-428f-b955-b2dec4471c5b-kube-api-access-965lz\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.288151 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.285688 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:39.878446 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.878416 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" event={"ID":"d26cecde-b16d-4fe1-add0-41a6d0a5301a","Type":"ContainerDied","Data":"fbfa0fa11cc19a966e687b7e5a97f6de9a180a3e08d5da107328232bfe3810d6"} Nov 04 01:34:39.878909 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.878496 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" event={"ID":"7619e54f-752b-428f-b955-b2dec4471c5b","Type":"ContainerDied","Data":"ee9904149a8a9ffb9c32a62ae8419c82a203d6f8b372329bc026fc83831b065b"} Nov 04 01:34:39.878909 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.878507 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr" Nov 04 01:34:39.878909 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.878510 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l" Nov 04 01:34:39.930709 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.930684 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr"] Nov 04 01:34:39.937310 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.937290 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-2ljkr"] Nov 04 01:34:39.948384 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.948359 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l"] Nov 04 01:34:39.953354 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.953336 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-w9f7l"] Nov 04 01:34:39.979800 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.979732 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:39.994117 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.994097 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7619e54f-752b-428f-b955-b2dec4471c5b" path="/var/lib/kubelet/pods/7619e54f-752b-428f-b955-b2dec4471c5b/volumes" Nov 04 01:34:39.994533 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:39.994519 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d26cecde-b16d-4fe1-add0-41a6d0a5301a" path="/var/lib/kubelet/pods/d26cecde-b16d-4fe1-add0-41a6d0a5301a/volumes" Nov 04 01:34:40.180283 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:40.180253 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:40.857475 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:40.857438 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:40.910789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:40.910753 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc" podUID="7af43160-27ea-4e98-8e54-46e9a0adc36c" containerName="payload-pre-processing" containerID="cri-o://4d30b0a9f7bda93093a0561d1eca450ef39d245c4a36484bb9679208b39b3a3f" gracePeriod=25 Nov 04 01:34:41.174441 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:41.174420 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc" Nov 04 01:34:41.276363 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:41.276332 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af43160-27ea-4e98-8e54-46e9a0adc36c-kube-api-access-vthzw\" (UniqueName: \"kubernetes.io/projected/7af43160-27ea-4e98-8e54-46e9a0adc36c-kube-api-access-vthzw\") pod \"7af43160-27ea-4e98-8e54-46e9a0adc36c\" (UID: \"7af43160-27ea-4e98-8e54-46e9a0adc36c\") " Nov 04 01:34:41.276519 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:41.276450 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7af43160-27ea-4e98-8e54-46e9a0adc36c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7af43160-27ea-4e98-8e54-46e9a0adc36c-plugins-config-volume\") pod \"7af43160-27ea-4e98-8e54-46e9a0adc36c\" (UID: \"7af43160-27ea-4e98-8e54-46e9a0adc36c\") " Nov 04 01:34:41.276827 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:41.276804 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af43160-27ea-4e98-8e54-46e9a0adc36c-plugins-config-volume" pod "7af43160-27ea-4e98-8e54-46e9a0adc36c" (UID: "7af43160-27ea-4e98-8e54-46e9a0adc36c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:41.277067 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:41.277020 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7af43160-27ea-4e98-8e54-46e9a0adc36c-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:41.278793 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:41.278765 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af43160-27ea-4e98-8e54-46e9a0adc36c-kube-api-access-vthzw" pod "7af43160-27ea-4e98-8e54-46e9a0adc36c" (UID: "7af43160-27ea-4e98-8e54-46e9a0adc36c"). InnerVolumeSpecName "kube-api-access-vthzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:41.378483 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:41.378452 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vthzw\" (UniqueName: \"kubernetes.io/projected/7af43160-27ea-4e98-8e54-46e9a0adc36c-kube-api-access-vthzw\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:41.623601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:41.620959 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:41.891063 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:41.890972 4852 generic.go:356] "Generic (PLEG): container finished" podID="7af43160-27ea-4e98-8e54-46e9a0adc36c" containerID="4d30b0a9f7bda93093a0561d1eca450ef39d245c4a36484bb9679208b39b3a3f" exitCode=0 Nov 04 01:34:41.891063 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:41.891020 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc" Nov 04 01:34:41.891259 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:41.891058 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc" event={"ID":"7af43160-27ea-4e98-8e54-46e9a0adc36c","Type":"ContainerDied","Data":"4d30b0a9f7bda93093a0561d1eca450ef39d245c4a36484bb9679208b39b3a3f"} Nov 04 01:34:41.891259 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:41.891131 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc" event={"ID":"7af43160-27ea-4e98-8e54-46e9a0adc36c","Type":"ContainerDied","Data":"66a9061a64de21f088789c5ba01b4320ac82cb004c58847b81a8f78a725c84e8"} Nov 04 01:34:41.891259 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:41.891158 4852 scope.go:122] "RemoveContainer" containerID="4d30b0a9f7bda93093a0561d1eca450ef39d245c4a36484bb9679208b39b3a3f" Nov 04 01:34:41.909262 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:41.909243 4852 scope.go:122] "RemoveContainer" containerID="4d30b0a9f7bda93093a0561d1eca450ef39d245c4a36484bb9679208b39b3a3f" Nov 04 01:34:41.909528 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:34:41.909509 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d30b0a9f7bda93093a0561d1eca450ef39d245c4a36484bb9679208b39b3a3f\": container with ID starting with 4d30b0a9f7bda93093a0561d1eca450ef39d245c4a36484bb9679208b39b3a3f not found: ID does not exist" containerID="4d30b0a9f7bda93093a0561d1eca450ef39d245c4a36484bb9679208b39b3a3f" Nov 04 01:34:41.909603 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:41.909535 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d30b0a9f7bda93093a0561d1eca450ef39d245c4a36484bb9679208b39b3a3f"} err="failed to get container status \"4d30b0a9f7bda93093a0561d1eca450ef39d245c4a36484bb9679208b39b3a3f\": rpc error: code = NotFound desc = could not find container \"4d30b0a9f7bda93093a0561d1eca450ef39d245c4a36484bb9679208b39b3a3f\": container with ID starting with 4d30b0a9f7bda93093a0561d1eca450ef39d245c4a36484bb9679208b39b3a3f not found: ID does not exist" Nov 04 01:34:41.935680 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:41.935657 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc"] Nov 04 01:34:41.945299 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:41.945276 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-98f6f6-7599f7df5-5kszc"] Nov 04 01:34:41.992187 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:41.992161 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:41.994685 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:41.994666 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7af43160-27ea-4e98-8e54-46e9a0adc36c" path="/var/lib/kubelet/pods/7af43160-27ea-4e98-8e54-46e9a0adc36c/volumes" Nov 04 01:34:42.146501 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:42.146405 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:42.510616 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:42.510083 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:42.974042 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:42.973831 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m" podUID="ed9eda72-0b3b-4933-bb4d-83d8eee5c20e" containerName="payload-processing" containerID="cri-o://e95d86cbff8b89fd9e939e914ce94b71f371892cfb6d8d7e2cbfbc38949a7883" gracePeriod=25 Nov 04 01:34:43.002905 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.002853 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb" podUID="e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed" containerName="payload-pre-processing" containerID="cri-o://b3ec36efa7be447a2c582bfb3a0963e666cbfb9a07c949c48ea57447a50fb881" gracePeriod=25 Nov 04 01:34:43.299807 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.299781 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb" Nov 04 01:34:43.348452 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.348422 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m" Nov 04 01:34:43.364275 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.364247 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx"] Nov 04 01:34:43.364499 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.364468 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" podUID="31ae07d8-c2d3-4732-83ff-b9583ec0fba8" containerName="maas-api" containerID="cri-o://6c460bccca45abb205209bea483d74d482bc5877b599b771613093bfbac3ba32" gracePeriod=30 Nov 04 01:34:43.435000 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.434964 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed-kube-api-access-g4k5m\" (UniqueName: \"kubernetes.io/projected/e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed-kube-api-access-g4k5m\") pod \"e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed\" (UID: \"e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed\") " Nov 04 01:34:43.435186 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.435038 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed9eda72-0b3b-4933-bb4d-83d8eee5c20e-kube-api-access-9lpn9\" (UniqueName: \"kubernetes.io/projected/ed9eda72-0b3b-4933-bb4d-83d8eee5c20e-kube-api-access-9lpn9\") pod \"ed9eda72-0b3b-4933-bb4d-83d8eee5c20e\" (UID: \"ed9eda72-0b3b-4933-bb4d-83d8eee5c20e\") " Nov 04 01:34:43.435186 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.435146 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed-plugins-config-volume\") pod \"e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed\" (UID: \"e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed\") " Nov 04 01:34:43.435310 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.435185 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed9eda72-0b3b-4933-bb4d-83d8eee5c20e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed9eda72-0b3b-4933-bb4d-83d8eee5c20e-plugins-config-volume\") pod \"ed9eda72-0b3b-4933-bb4d-83d8eee5c20e\" (UID: \"ed9eda72-0b3b-4933-bb4d-83d8eee5c20e\") " Nov 04 01:34:43.435982 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.435950 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed9eda72-0b3b-4933-bb4d-83d8eee5c20e-plugins-config-volume" pod "ed9eda72-0b3b-4933-bb4d-83d8eee5c20e" (UID: "ed9eda72-0b3b-4933-bb4d-83d8eee5c20e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:43.436327 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.436308 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed-plugins-config-volume" pod "e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed" (UID: "e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:43.437955 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.437922 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9eda72-0b3b-4933-bb4d-83d8eee5c20e-kube-api-access-9lpn9" pod "ed9eda72-0b3b-4933-bb4d-83d8eee5c20e" (UID: "ed9eda72-0b3b-4933-bb4d-83d8eee5c20e"). InnerVolumeSpecName "kube-api-access-9lpn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:43.438406 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.438377 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed-kube-api-access-g4k5m" pod "e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed" (UID: "e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed"). InnerVolumeSpecName "kube-api-access-g4k5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:43.483106 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.483024 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:43.536550 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.536514 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4k5m\" (UniqueName: \"kubernetes.io/projected/e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed-kube-api-access-g4k5m\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:43.536550 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.536546 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lpn9\" (UniqueName: \"kubernetes.io/projected/ed9eda72-0b3b-4933-bb4d-83d8eee5c20e-kube-api-access-9lpn9\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:43.536550 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.536556 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:43.536550 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.536586 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed9eda72-0b3b-4933-bb4d-83d8eee5c20e-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:43.706352 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.706056 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" Nov 04 01:34:43.866586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.859040 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-maas-api-tls\") pod \"31ae07d8-c2d3-4732-83ff-b9583ec0fba8\" (UID: \"31ae07d8-c2d3-4732-83ff-b9583ec0fba8\") " Nov 04 01:34:43.866586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.859223 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-openshift-service-ca\") pod \"31ae07d8-c2d3-4732-83ff-b9583ec0fba8\" (UID: \"31ae07d8-c2d3-4732-83ff-b9583ec0fba8\") " Nov 04 01:34:43.866586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.859372 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-kube-api-access-dxcz7\" (UniqueName: \"kubernetes.io/projected/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-kube-api-access-dxcz7\") pod \"31ae07d8-c2d3-4732-83ff-b9583ec0fba8\" (UID: \"31ae07d8-c2d3-4732-83ff-b9583ec0fba8\") " Nov 04 01:34:43.866586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.859608 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-metrics-certs\") pod \"31ae07d8-c2d3-4732-83ff-b9583ec0fba8\" (UID: \"31ae07d8-c2d3-4732-83ff-b9583ec0fba8\") " Nov 04 01:34:43.888542 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.888492 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-metrics-certs" pod "31ae07d8-c2d3-4732-83ff-b9583ec0fba8" (UID: "31ae07d8-c2d3-4732-83ff-b9583ec0fba8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:43.889249 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.889218 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-openshift-service-ca" pod "31ae07d8-c2d3-4732-83ff-b9583ec0fba8" (UID: "31ae07d8-c2d3-4732-83ff-b9583ec0fba8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:43.890449 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.890414 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:43.892606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.892540 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-maas-api-tls" pod "31ae07d8-c2d3-4732-83ff-b9583ec0fba8" (UID: "31ae07d8-c2d3-4732-83ff-b9583ec0fba8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:43.892721 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.892622 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-kube-api-access-dxcz7" pod "31ae07d8-c2d3-4732-83ff-b9583ec0fba8" (UID: "31ae07d8-c2d3-4732-83ff-b9583ec0fba8"). InnerVolumeSpecName "kube-api-access-dxcz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:43.912408 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.912378 4852 generic.go:356] "Generic (PLEG): container finished" podID="ed9eda72-0b3b-4933-bb4d-83d8eee5c20e" containerID="e95d86cbff8b89fd9e939e914ce94b71f371892cfb6d8d7e2cbfbc38949a7883" exitCode=0 Nov 04 01:34:43.912627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.912428 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m" Nov 04 01:34:43.912627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.912463 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m" event={"ID":"ed9eda72-0b3b-4933-bb4d-83d8eee5c20e","Type":"ContainerDied","Data":"e95d86cbff8b89fd9e939e914ce94b71f371892cfb6d8d7e2cbfbc38949a7883"} Nov 04 01:34:43.912627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.912516 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m" event={"ID":"ed9eda72-0b3b-4933-bb4d-83d8eee5c20e","Type":"ContainerDied","Data":"8d5900de47eef8558fc223f1940913fab407629d0040a31c11fc16b4a4e5c86a"} Nov 04 01:34:43.912627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.912543 4852 scope.go:122] "RemoveContainer" containerID="e95d86cbff8b89fd9e939e914ce94b71f371892cfb6d8d7e2cbfbc38949a7883" Nov 04 01:34:43.919590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.913988 4852 generic.go:356] "Generic (PLEG): container finished" podID="e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed" containerID="b3ec36efa7be447a2c582bfb3a0963e666cbfb9a07c949c48ea57447a50fb881" exitCode=0 Nov 04 01:34:43.919590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.914022 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb" event={"ID":"e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed","Type":"ContainerDied","Data":"b3ec36efa7be447a2c582bfb3a0963e666cbfb9a07c949c48ea57447a50fb881"} Nov 04 01:34:43.919590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.914046 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb" Nov 04 01:34:43.919590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.914121 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb" event={"ID":"e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed","Type":"ContainerDied","Data":"732faf5c87d699daa93a0d82ae0366b54fda90efaa245787e65f67832036cb1c"} Nov 04 01:34:43.919590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.915815 4852 generic.go:356] "Generic (PLEG): container finished" podID="31ae07d8-c2d3-4732-83ff-b9583ec0fba8" containerID="6c460bccca45abb205209bea483d74d482bc5877b599b771613093bfbac3ba32" exitCode=0 Nov 04 01:34:43.919590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.915889 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" Nov 04 01:34:43.919590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.915908 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" event={"ID":"31ae07d8-c2d3-4732-83ff-b9583ec0fba8","Type":"ContainerDied","Data":"6c460bccca45abb205209bea483d74d482bc5877b599b771613093bfbac3ba32"} Nov 04 01:34:43.919590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.915947 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx" event={"ID":"31ae07d8-c2d3-4732-83ff-b9583ec0fba8","Type":"ContainerDied","Data":"710419098e3dc4f587e2fcc50325278df9bd9061bcdddc20a96d6ec6711e630c"} Nov 04 01:34:43.926025 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.926006 4852 scope.go:122] "RemoveContainer" containerID="e95d86cbff8b89fd9e939e914ce94b71f371892cfb6d8d7e2cbfbc38949a7883" Nov 04 01:34:43.926368 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:34:43.926317 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95d86cbff8b89fd9e939e914ce94b71f371892cfb6d8d7e2cbfbc38949a7883\": container with ID starting with e95d86cbff8b89fd9e939e914ce94b71f371892cfb6d8d7e2cbfbc38949a7883 not found: ID does not exist" containerID="e95d86cbff8b89fd9e939e914ce94b71f371892cfb6d8d7e2cbfbc38949a7883" Nov 04 01:34:43.926368 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.926346 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95d86cbff8b89fd9e939e914ce94b71f371892cfb6d8d7e2cbfbc38949a7883"} err="failed to get container status \"e95d86cbff8b89fd9e939e914ce94b71f371892cfb6d8d7e2cbfbc38949a7883\": rpc error: code = NotFound desc = could not find container \"e95d86cbff8b89fd9e939e914ce94b71f371892cfb6d8d7e2cbfbc38949a7883\": container with ID starting with e95d86cbff8b89fd9e939e914ce94b71f371892cfb6d8d7e2cbfbc38949a7883 not found: ID does not exist" Nov 04 01:34:43.926368 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.926362 4852 scope.go:122] "RemoveContainer" containerID="b3ec36efa7be447a2c582bfb3a0963e666cbfb9a07c949c48ea57447a50fb881" Nov 04 01:34:43.934869 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.934850 4852 scope.go:122] "RemoveContainer" containerID="b3ec36efa7be447a2c582bfb3a0963e666cbfb9a07c949c48ea57447a50fb881" Nov 04 01:34:43.935132 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:34:43.935115 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3ec36efa7be447a2c582bfb3a0963e666cbfb9a07c949c48ea57447a50fb881\": container with ID starting with b3ec36efa7be447a2c582bfb3a0963e666cbfb9a07c949c48ea57447a50fb881 not found: ID does not exist" containerID="b3ec36efa7be447a2c582bfb3a0963e666cbfb9a07c949c48ea57447a50fb881" Nov 04 01:34:43.935181 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.935138 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3ec36efa7be447a2c582bfb3a0963e666cbfb9a07c949c48ea57447a50fb881"} err="failed to get container status \"b3ec36efa7be447a2c582bfb3a0963e666cbfb9a07c949c48ea57447a50fb881\": rpc error: code = NotFound desc = could not find container \"b3ec36efa7be447a2c582bfb3a0963e666cbfb9a07c949c48ea57447a50fb881\": container with ID starting with b3ec36efa7be447a2c582bfb3a0963e666cbfb9a07c949c48ea57447a50fb881 not found: ID does not exist" Nov 04 01:34:43.935181 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.935152 4852 scope.go:122] "RemoveContainer" containerID="6c460bccca45abb205209bea483d74d482bc5877b599b771613093bfbac3ba32" Nov 04 01:34:43.941883 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.941866 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m"] Nov 04 01:34:43.948473 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.947298 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-af6909-dc65cbbb-7dt7m"] Nov 04 01:34:43.949668 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.949641 4852 scope.go:122] "RemoveContainer" containerID="6c460bccca45abb205209bea483d74d482bc5877b599b771613093bfbac3ba32" Nov 04 01:34:43.949948 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:34:43.949928 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c460bccca45abb205209bea483d74d482bc5877b599b771613093bfbac3ba32\": container with ID starting with 6c460bccca45abb205209bea483d74d482bc5877b599b771613093bfbac3ba32 not found: ID does not exist" containerID="6c460bccca45abb205209bea483d74d482bc5877b599b771613093bfbac3ba32" Nov 04 01:34:43.950012 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.949952 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c460bccca45abb205209bea483d74d482bc5877b599b771613093bfbac3ba32"} err="failed to get container status \"6c460bccca45abb205209bea483d74d482bc5877b599b771613093bfbac3ba32\": rpc error: code = NotFound desc = could not find container \"6c460bccca45abb205209bea483d74d482bc5877b599b771613093bfbac3ba32\": container with ID starting with 6c460bccca45abb205209bea483d74d482bc5877b599b771613093bfbac3ba32 not found: ID does not exist" Nov 04 01:34:43.961296 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.961272 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxcz7\" (UniqueName: \"kubernetes.io/projected/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-kube-api-access-dxcz7\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:43.961372 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.961301 4852 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-metrics-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:43.961372 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.961316 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-maas-api-tls\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:43.961372 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.961331 4852 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ae07d8-c2d3-4732-83ff-b9583ec0fba8-openshift-service-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:43.962099 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.962081 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb"] Nov 04 01:34:43.967554 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.967530 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-af6909-fbcccbbc9-sjqcb"] Nov 04 01:34:43.977684 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.977655 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx"] Nov 04 01:34:43.984109 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.984088 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-af6909-6f957d984f-jfrrx"] Nov 04 01:34:43.993484 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.993454 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31ae07d8-c2d3-4732-83ff-b9583ec0fba8" path="/var/lib/kubelet/pods/31ae07d8-c2d3-4732-83ff-b9583ec0fba8/volumes" Nov 04 01:34:43.993917 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.993898 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed" path="/var/lib/kubelet/pods/e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed/volumes" Nov 04 01:34:43.994310 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:43.994292 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed9eda72-0b3b-4933-bb4d-83d8eee5c20e" path="/var/lib/kubelet/pods/ed9eda72-0b3b-4933-bb4d-83d8eee5c20e/volumes" Nov 04 01:34:44.131542 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:44.131482 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx"] Nov 04 01:34:44.131714 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:44.131694 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" podUID="719d829c-c70a-4bdc-a3d2-5b4cb52a45da" containerName="istio-proxy" containerID="cri-o://d1ad05404360bac0238027aa3d12aeff93e4b892302fa5a0de5acc3c1c2ac70b" gracePeriod=30 Nov 04 01:34:44.711502 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:44.711470 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:44.882385 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:44.882355 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:45.448979 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:45.448941 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:45.962606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:45.959369 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:46.188172 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:46.188096 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:46.220733 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:46.220616 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g"] Nov 04 01:34:46.234608 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:46.234552 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv"] Nov 04 01:34:46.356361 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:46.356331 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:46.771861 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:46.771828 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:46.832383 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:46.827155 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4"] Nov 04 01:34:46.832383 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:46.827381 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" containerID="cri-o://a702664a3fa1c8f8d72c93af6783b851dd5ce5b09759f0922c7efc0a012b5979" gracePeriod=30 Nov 04 01:34:46.998334 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:46.998069 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:47.320690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:47.320660 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:47.480772 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:47.480735 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:47.720283 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:47.720201 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:48.062275 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:48.062226 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:48.406275 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:48.400953 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:48.573849 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:48.573812 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:48.747624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:48.747519 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:48.922655 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:48.922342 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:49.193721 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.193671 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f"] Nov 04 01:34:49.194187 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.193930 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" podUID="d31799de-2b0d-4f2e-839e-058668f304c6" containerName="istio-proxy" containerID="cri-o://c617da0f9451141112c806b4771869ea40c51f980e5bb4b1f7b207d843c5e4f2" gracePeriod=30 Nov 04 01:34:49.289961 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.289899 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:49.422960 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.422934 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:34:49.551591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.547029 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:49.566589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.563798 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-podinfo\") pod \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " Nov 04 01:34:49.566589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.563884 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-kube-api-access-bgzsb\" (UniqueName: \"kubernetes.io/projected/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-kube-api-access-bgzsb\") pod \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " Nov 04 01:34:49.566589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.563920 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-envoy\") pod \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " Nov 04 01:34:49.566589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.563956 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istiod-ca-cert\") pod \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " Nov 04 01:34:49.566589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.564018 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-workload-certs\") pod \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " Nov 04 01:34:49.566589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.564158 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-workload-socket\") pod \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " Nov 04 01:34:49.566589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.564194 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-credential-socket\") pod \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " Nov 04 01:34:49.566589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.564243 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-token\" (UniqueName: \"kubernetes.io/projected/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-token\") pod \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " Nov 04 01:34:49.566589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.564278 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-data\") pod \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\" (UID: \"719d829c-c70a-4bdc-a3d2-5b4cb52a45da\") " Nov 04 01:34:49.566589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.565207 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-data" pod "719d829c-c70a-4bdc-a3d2-5b4cb52a45da" (UID: "719d829c-c70a-4bdc-a3d2-5b4cb52a45da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:49.566589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.565792 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-workload-certs" pod "719d829c-c70a-4bdc-a3d2-5b4cb52a45da" (UID: "719d829c-c70a-4bdc-a3d2-5b4cb52a45da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:49.570338 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.568511 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-kube-api-access-bgzsb" pod "719d829c-c70a-4bdc-a3d2-5b4cb52a45da" (UID: "719d829c-c70a-4bdc-a3d2-5b4cb52a45da"). InnerVolumeSpecName "kube-api-access-bgzsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:49.591934 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.581013 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-podinfo" pod "719d829c-c70a-4bdc-a3d2-5b4cb52a45da" (UID: "719d829c-c70a-4bdc-a3d2-5b4cb52a45da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:49.591934 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.581219 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-workload-socket" pod "719d829c-c70a-4bdc-a3d2-5b4cb52a45da" (UID: "719d829c-c70a-4bdc-a3d2-5b4cb52a45da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:49.591934 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.581388 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-credential-socket" pod "719d829c-c70a-4bdc-a3d2-5b4cb52a45da" (UID: "719d829c-c70a-4bdc-a3d2-5b4cb52a45da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:49.591934 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.582049 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-envoy" pod "719d829c-c70a-4bdc-a3d2-5b4cb52a45da" (UID: "719d829c-c70a-4bdc-a3d2-5b4cb52a45da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:49.591934 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.591765 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istiod-ca-cert" pod "719d829c-c70a-4bdc-a3d2-5b4cb52a45da" (UID: "719d829c-c70a-4bdc-a3d2-5b4cb52a45da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:49.651962 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.651871 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-token" pod "719d829c-c70a-4bdc-a3d2-5b4cb52a45da" (UID: "719d829c-c70a-4bdc-a3d2-5b4cb52a45da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:49.678001 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.677967 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:49.678001 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.678000 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:49.682229 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.678013 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:49.682229 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.678025 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:49.682229 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.678037 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:49.682229 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.678048 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgzsb\" (UniqueName: \"kubernetes.io/projected/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-kube-api-access-bgzsb\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:49.682229 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.678062 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:49.682229 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.678073 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:49.682229 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.678084 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/719d829c-c70a-4bdc-a3d2-5b4cb52a45da-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:49.853784 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:49.853751 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:50.006433 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:50.006407 4852 generic.go:356] "Generic (PLEG): container finished" podID="719d829c-c70a-4bdc-a3d2-5b4cb52a45da" containerID="d1ad05404360bac0238027aa3d12aeff93e4b892302fa5a0de5acc3c1c2ac70b" exitCode=0 Nov 04 01:34:50.006557 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:50.006443 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" Nov 04 01:34:50.006557 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:50.006491 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" event={"ID":"719d829c-c70a-4bdc-a3d2-5b4cb52a45da","Type":"ContainerDied","Data":"d1ad05404360bac0238027aa3d12aeff93e4b892302fa5a0de5acc3c1c2ac70b"} Nov 04 01:34:50.006557 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:50.006539 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx" event={"ID":"719d829c-c70a-4bdc-a3d2-5b4cb52a45da","Type":"ContainerDied","Data":"804f4a4597786e809539f92397e46d3fa65b3450a47a8a17f7690887fb30f962"} Nov 04 01:34:50.006738 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:50.006600 4852 scope.go:122] "RemoveContainer" containerID="d1ad05404360bac0238027aa3d12aeff93e4b892302fa5a0de5acc3c1c2ac70b" Nov 04 01:34:50.016018 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:50.015995 4852 scope.go:122] "RemoveContainer" containerID="d1ad05404360bac0238027aa3d12aeff93e4b892302fa5a0de5acc3c1c2ac70b" Nov 04 01:34:50.016286 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:34:50.016259 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ad05404360bac0238027aa3d12aeff93e4b892302fa5a0de5acc3c1c2ac70b\": container with ID starting with d1ad05404360bac0238027aa3d12aeff93e4b892302fa5a0de5acc3c1c2ac70b not found: ID does not exist" containerID="d1ad05404360bac0238027aa3d12aeff93e4b892302fa5a0de5acc3c1c2ac70b" Nov 04 01:34:50.016354 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:50.016297 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ad05404360bac0238027aa3d12aeff93e4b892302fa5a0de5acc3c1c2ac70b"} err="failed to get container status \"d1ad05404360bac0238027aa3d12aeff93e4b892302fa5a0de5acc3c1c2ac70b\": rpc error: code = NotFound desc = could not find container \"d1ad05404360bac0238027aa3d12aeff93e4b892302fa5a0de5acc3c1c2ac70b\": container with ID starting with d1ad05404360bac0238027aa3d12aeff93e4b892302fa5a0de5acc3c1c2ac70b not found: ID does not exist" Nov 04 01:34:50.032966 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:50.032941 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx"] Nov 04 01:34:50.036098 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:50.036078 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-d5bbx"] Nov 04 01:34:50.170694 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:50.170668 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:50.893736 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:50.887076 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:51.222148 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:51.222064 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g" podUID="39133c56-c8e4-4b42-9df6-7dc091b4f09d" containerName="payload-processing" containerID="cri-o://ce0a204c3e522d9d0e14fcc69b0bdb1f08dd639cf3b142b5d3428d0e795ae197" gracePeriod=25 Nov 04 01:34:51.235629 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:51.235601 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv" podUID="0e92e697-d5a8-4f9c-9716-697d3f9e2546" containerName="payload-pre-processing" containerID="cri-o://4e9c4129112966b94dde1fb82a86d10a71a295309b5ee5d3731f5bf441387952" gracePeriod=25 Nov 04 01:34:51.496967 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:51.496942 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g" Nov 04 01:34:51.517754 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:51.517728 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv" Nov 04 01:34:51.600248 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:51.600219 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39133c56-c8e4-4b42-9df6-7dc091b4f09d-kube-api-access-rhfdw\" (UniqueName: \"kubernetes.io/projected/39133c56-c8e4-4b42-9df6-7dc091b4f09d-kube-api-access-rhfdw\") pod \"39133c56-c8e4-4b42-9df6-7dc091b4f09d\" (UID: \"39133c56-c8e4-4b42-9df6-7dc091b4f09d\") " Nov 04 01:34:51.600394 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:51.600304 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39133c56-c8e4-4b42-9df6-7dc091b4f09d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39133c56-c8e4-4b42-9df6-7dc091b4f09d-plugins-config-volume\") pod \"39133c56-c8e4-4b42-9df6-7dc091b4f09d\" (UID: \"39133c56-c8e4-4b42-9df6-7dc091b4f09d\") " Nov 04 01:34:51.600394 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:51.600352 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e92e697-d5a8-4f9c-9716-697d3f9e2546-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e92e697-d5a8-4f9c-9716-697d3f9e2546-plugins-config-volume\") pod \"0e92e697-d5a8-4f9c-9716-697d3f9e2546\" (UID: \"0e92e697-d5a8-4f9c-9716-697d3f9e2546\") " Nov 04 01:34:51.600394 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:51.600373 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e92e697-d5a8-4f9c-9716-697d3f9e2546-kube-api-access-cddwc\" (UniqueName: \"kubernetes.io/projected/0e92e697-d5a8-4f9c-9716-697d3f9e2546-kube-api-access-cddwc\") pod \"0e92e697-d5a8-4f9c-9716-697d3f9e2546\" (UID: \"0e92e697-d5a8-4f9c-9716-697d3f9e2546\") " Nov 04 01:34:51.600709 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:51.600686 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39133c56-c8e4-4b42-9df6-7dc091b4f09d-plugins-config-volume" pod "39133c56-c8e4-4b42-9df6-7dc091b4f09d" (UID: "39133c56-c8e4-4b42-9df6-7dc091b4f09d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:51.600790 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:51.600755 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e92e697-d5a8-4f9c-9716-697d3f9e2546-plugins-config-volume" pod "0e92e697-d5a8-4f9c-9716-697d3f9e2546" (UID: "0e92e697-d5a8-4f9c-9716-697d3f9e2546"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:51.601024 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:51.600971 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39133c56-c8e4-4b42-9df6-7dc091b4f09d-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:51.601024 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:51.600991 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e92e697-d5a8-4f9c-9716-697d3f9e2546-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:51.602552 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:51.602529 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e92e697-d5a8-4f9c-9716-697d3f9e2546-kube-api-access-cddwc" pod "0e92e697-d5a8-4f9c-9716-697d3f9e2546" (UID: "0e92e697-d5a8-4f9c-9716-697d3f9e2546"). InnerVolumeSpecName "kube-api-access-cddwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:51.602661 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:51.602593 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39133c56-c8e4-4b42-9df6-7dc091b4f09d-kube-api-access-rhfdw" pod "39133c56-c8e4-4b42-9df6-7dc091b4f09d" (UID: "39133c56-c8e4-4b42-9df6-7dc091b4f09d"). InnerVolumeSpecName "kube-api-access-rhfdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:51.681969 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:51.681912 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:51.702651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:51.702619 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhfdw\" (UniqueName: \"kubernetes.io/projected/39133c56-c8e4-4b42-9df6-7dc091b4f09d-kube-api-access-rhfdw\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:51.702651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:51.702646 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cddwc\" (UniqueName: \"kubernetes.io/projected/0e92e697-d5a8-4f9c-9716-697d3f9e2546-kube-api-access-cddwc\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:51.994764 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:51.994731 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="719d829c-c70a-4bdc-a3d2-5b4cb52a45da" path="/var/lib/kubelet/pods/719d829c-c70a-4bdc-a3d2-5b4cb52a45da/volumes" Nov 04 01:34:52.021391 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.021360 4852 generic.go:356] "Generic (PLEG): container finished" podID="0e92e697-d5a8-4f9c-9716-697d3f9e2546" containerID="4e9c4129112966b94dde1fb82a86d10a71a295309b5ee5d3731f5bf441387952" exitCode=0 Nov 04 01:34:52.021512 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.021405 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv" Nov 04 01:34:52.021512 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.021446 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv" event={"ID":"0e92e697-d5a8-4f9c-9716-697d3f9e2546","Type":"ContainerDied","Data":"4e9c4129112966b94dde1fb82a86d10a71a295309b5ee5d3731f5bf441387952"} Nov 04 01:34:52.021692 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.021539 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv" event={"ID":"0e92e697-d5a8-4f9c-9716-697d3f9e2546","Type":"ContainerDied","Data":"62a57276ed7106e62743db74206ecc11da972fee8ae045eeacb87a2358124530"} Nov 04 01:34:52.021692 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.021612 4852 scope.go:122] "RemoveContainer" containerID="4e9c4129112966b94dde1fb82a86d10a71a295309b5ee5d3731f5bf441387952" Nov 04 01:34:52.023939 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.023921 4852 generic.go:356] "Generic (PLEG): container finished" podID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerID="a702664a3fa1c8f8d72c93af6783b851dd5ce5b09759f0922c7efc0a012b5979" exitCode=0 Nov 04 01:34:52.024040 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.023996 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" event={"ID":"c72aca29-e3f2-4a8f-abea-8841337cc569","Type":"ContainerDied","Data":"a702664a3fa1c8f8d72c93af6783b851dd5ce5b09759f0922c7efc0a012b5979"} Nov 04 01:34:52.025525 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.025501 4852 generic.go:356] "Generic (PLEG): container finished" podID="39133c56-c8e4-4b42-9df6-7dc091b4f09d" containerID="ce0a204c3e522d9d0e14fcc69b0bdb1f08dd639cf3b142b5d3428d0e795ae197" exitCode=0 Nov 04 01:34:52.025647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.025545 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g" Nov 04 01:34:52.025647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.025599 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g" event={"ID":"39133c56-c8e4-4b42-9df6-7dc091b4f09d","Type":"ContainerDied","Data":"ce0a204c3e522d9d0e14fcc69b0bdb1f08dd639cf3b142b5d3428d0e795ae197"} Nov 04 01:34:52.025647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.025633 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g" event={"ID":"39133c56-c8e4-4b42-9df6-7dc091b4f09d","Type":"ContainerDied","Data":"6f252deab79ed469903186cf4a3dd87f3c8185062080c9098ec326c258f012a9"} Nov 04 01:34:52.032078 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.032061 4852 scope.go:122] "RemoveContainer" containerID="4e9c4129112966b94dde1fb82a86d10a71a295309b5ee5d3731f5bf441387952" Nov 04 01:34:52.032299 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:34:52.032284 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e9c4129112966b94dde1fb82a86d10a71a295309b5ee5d3731f5bf441387952\": container with ID starting with 4e9c4129112966b94dde1fb82a86d10a71a295309b5ee5d3731f5bf441387952 not found: ID does not exist" containerID="4e9c4129112966b94dde1fb82a86d10a71a295309b5ee5d3731f5bf441387952" Nov 04 01:34:52.032348 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.032305 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e9c4129112966b94dde1fb82a86d10a71a295309b5ee5d3731f5bf441387952"} err="failed to get container status \"4e9c4129112966b94dde1fb82a86d10a71a295309b5ee5d3731f5bf441387952\": rpc error: code = NotFound desc = could not find container \"4e9c4129112966b94dde1fb82a86d10a71a295309b5ee5d3731f5bf441387952\": container with ID starting with 4e9c4129112966b94dde1fb82a86d10a71a295309b5ee5d3731f5bf441387952 not found: ID does not exist" Nov 04 01:34:52.032348 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.032319 4852 scope.go:122] "RemoveContainer" containerID="25d8e1ca785c573a8219c4cfff8b56925602c99c8b31a7953c1d3f17e234ac6e" Nov 04 01:34:52.038814 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.038794 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv"] Nov 04 01:34:52.043528 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.043503 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dfcebf-644979847-zrrmv"] Nov 04 01:34:52.045151 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.045136 4852 scope.go:122] "RemoveContainer" containerID="ce0a204c3e522d9d0e14fcc69b0bdb1f08dd639cf3b142b5d3428d0e795ae197" Nov 04 01:34:52.053073 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.053051 4852 scope.go:122] "RemoveContainer" containerID="ce0a204c3e522d9d0e14fcc69b0bdb1f08dd639cf3b142b5d3428d0e795ae197" Nov 04 01:34:52.053498 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:34:52.053339 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce0a204c3e522d9d0e14fcc69b0bdb1f08dd639cf3b142b5d3428d0e795ae197\": container with ID starting with ce0a204c3e522d9d0e14fcc69b0bdb1f08dd639cf3b142b5d3428d0e795ae197 not found: ID does not exist" containerID="ce0a204c3e522d9d0e14fcc69b0bdb1f08dd639cf3b142b5d3428d0e795ae197" Nov 04 01:34:52.053608 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.053511 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce0a204c3e522d9d0e14fcc69b0bdb1f08dd639cf3b142b5d3428d0e795ae197"} err="failed to get container status \"ce0a204c3e522d9d0e14fcc69b0bdb1f08dd639cf3b142b5d3428d0e795ae197\": rpc error: code = NotFound desc = could not find container \"ce0a204c3e522d9d0e14fcc69b0bdb1f08dd639cf3b142b5d3428d0e795ae197\": container with ID starting with ce0a204c3e522d9d0e14fcc69b0bdb1f08dd639cf3b142b5d3428d0e795ae197 not found: ID does not exist" Nov 04 01:34:52.055668 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.055646 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g"] Nov 04 01:34:52.058746 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.058727 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dfcebf-65f84c6744-2zz6g"] Nov 04 01:34:52.371411 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.370781 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:52.391259 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.391228 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx"] Nov 04 01:34:52.399788 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.399763 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94"] Nov 04 01:34:52.529862 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.529839 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:34:52.537492 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.537470 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:52.661913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.627128 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c72aca29-e3f2-4a8f-abea-8841337cc569-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c72aca29-e3f2-4a8f-abea-8841337cc569-istiod-ca-cert\") pod \"c72aca29-e3f2-4a8f-abea-8841337cc569\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " Nov 04 01:34:52.661913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.627614 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c72aca29-e3f2-4a8f-abea-8841337cc569-istiod-ca-cert" pod "c72aca29-e3f2-4a8f-abea-8841337cc569" (UID: "c72aca29-e3f2-4a8f-abea-8841337cc569"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:52.661913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.627696 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c72aca29-e3f2-4a8f-abea-8841337cc569-kube-api-access-r9ln9\" (UniqueName: \"kubernetes.io/projected/c72aca29-e3f2-4a8f-abea-8841337cc569-kube-api-access-r9ln9\") pod \"c72aca29-e3f2-4a8f-abea-8841337cc569\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " Nov 04 01:34:52.661913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.628340 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-envoy\") pod \"c72aca29-e3f2-4a8f-abea-8841337cc569\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " Nov 04 01:34:52.661913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.629072 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-workload-certs\") pod \"c72aca29-e3f2-4a8f-abea-8841337cc569\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " Nov 04 01:34:52.661913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.629417 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-workload-certs" pod "c72aca29-e3f2-4a8f-abea-8841337cc569" (UID: "c72aca29-e3f2-4a8f-abea-8841337cc569"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:52.661913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.629540 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-credential-socket\") pod \"c72aca29-e3f2-4a8f-abea-8841337cc569\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " Nov 04 01:34:52.661913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.630965 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-credential-socket" pod "c72aca29-e3f2-4a8f-abea-8841337cc569" (UID: "c72aca29-e3f2-4a8f-abea-8841337cc569"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:52.661913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.638647 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-data" pod "c72aca29-e3f2-4a8f-abea-8841337cc569" (UID: "c72aca29-e3f2-4a8f-abea-8841337cc569"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:52.661913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.638897 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-data\") pod \"c72aca29-e3f2-4a8f-abea-8841337cc569\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " Nov 04 01:34:52.661913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.653445 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-workload-socket\") pod \"c72aca29-e3f2-4a8f-abea-8841337cc569\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " Nov 04 01:34:52.661913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.653531 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-podinfo\") pod \"c72aca29-e3f2-4a8f-abea-8841337cc569\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " Nov 04 01:34:52.661913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.653581 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-token\" (UniqueName: \"kubernetes.io/projected/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-token\") pod \"c72aca29-e3f2-4a8f-abea-8841337cc569\" (UID: \"c72aca29-e3f2-4a8f-abea-8841337cc569\") " Nov 04 01:34:52.663592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.663322 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-workload-socket" pod "c72aca29-e3f2-4a8f-abea-8841337cc569" (UID: "c72aca29-e3f2-4a8f-abea-8841337cc569"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:52.663592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.663489 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72aca29-e3f2-4a8f-abea-8841337cc569-kube-api-access-r9ln9" pod "c72aca29-e3f2-4a8f-abea-8841337cc569" (UID: "c72aca29-e3f2-4a8f-abea-8841337cc569"). InnerVolumeSpecName "kube-api-access-r9ln9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:52.663592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.663552 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-envoy" pod "c72aca29-e3f2-4a8f-abea-8841337cc569" (UID: "c72aca29-e3f2-4a8f-abea-8841337cc569"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:52.666593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.664943 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:52.666593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.664975 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:52.666593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.664994 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:52.666593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.665017 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c72aca29-e3f2-4a8f-abea-8841337cc569-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:52.666593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.665207 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9ln9\" (UniqueName: \"kubernetes.io/projected/c72aca29-e3f2-4a8f-abea-8841337cc569-kube-api-access-r9ln9\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:52.666593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.665222 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:52.666593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.665237 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c72aca29-e3f2-4a8f-abea-8841337cc569-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:52.806931 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.806776 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:52.936591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.934389 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-podinfo" pod "c72aca29-e3f2-4a8f-abea-8841337cc569" (UID: "c72aca29-e3f2-4a8f-abea-8841337cc569"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:52.950936 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.937738 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-token" pod "c72aca29-e3f2-4a8f-abea-8841337cc569" (UID: "c72aca29-e3f2-4a8f-abea-8841337cc569"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:52.987931 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.987893 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:52.988176 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:52.988150 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c72aca29-e3f2-4a8f-abea-8841337cc569-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:53.043601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:53.041231 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" Nov 04 01:34:53.043601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:53.041242 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4" event={"ID":"c72aca29-e3f2-4a8f-abea-8841337cc569","Type":"ContainerDied","Data":"ce0d5542f158588e32a825e4b37813b1525a1f43894e4b4c9ab36fe460ac7de4"} Nov 04 01:34:53.043601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:53.041319 4852 scope.go:122] "RemoveContainer" containerID="a702664a3fa1c8f8d72c93af6783b851dd5ce5b09759f0922c7efc0a012b5979" Nov 04 01:34:53.047256 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:53.047228 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:53.066793 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:53.066737 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4"] Nov 04 01:34:53.072888 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:53.072863 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-98f6f6-openshift-default-6f559885c-jhzs4"] Nov 04 01:34:53.193547 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:53.193512 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:53.348356 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:53.348271 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:53.995308 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:53.995274 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e92e697-d5a8-4f9c-9716-697d3f9e2546" path="/var/lib/kubelet/pods/0e92e697-d5a8-4f9c-9716-697d3f9e2546/volumes" Nov 04 01:34:53.995681 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:53.995666 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39133c56-c8e4-4b42-9df6-7dc091b4f09d" path="/var/lib/kubelet/pods/39133c56-c8e4-4b42-9df6-7dc091b4f09d/volumes" Nov 04 01:34:53.996074 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:53.996061 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" path="/var/lib/kubelet/pods/c72aca29-e3f2-4a8f-abea-8841337cc569/volumes" Nov 04 01:34:54.149380 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.149309 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:54.395455 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.395421 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:54.443322 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.443302 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:34:54.507950 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.507923 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d31799de-2b0d-4f2e-839e-058668f304c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31799de-2b0d-4f2e-839e-058668f304c6-istio-podinfo\") pod \"d31799de-2b0d-4f2e-839e-058668f304c6\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " Nov 04 01:34:54.508091 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.507993 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-istio-envoy\") pod \"d31799de-2b0d-4f2e-839e-058668f304c6\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " Nov 04 01:34:54.508091 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.508028 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31799de-2b0d-4f2e-839e-058668f304c6-istio-token\" (UniqueName: \"kubernetes.io/projected/d31799de-2b0d-4f2e-839e-058668f304c6-istio-token\") pod \"d31799de-2b0d-4f2e-839e-058668f304c6\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " Nov 04 01:34:54.508091 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.508083 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d31799de-2b0d-4f2e-839e-058668f304c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31799de-2b0d-4f2e-839e-058668f304c6-istiod-ca-cert\") pod \"d31799de-2b0d-4f2e-839e-058668f304c6\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " Nov 04 01:34:54.508276 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.508156 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-credential-socket\") pod \"d31799de-2b0d-4f2e-839e-058668f304c6\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " Nov 04 01:34:54.508276 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.508206 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-workload-socket\") pod \"d31799de-2b0d-4f2e-839e-058668f304c6\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " Nov 04 01:34:54.508276 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.508260 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-workload-certs\") pod \"d31799de-2b0d-4f2e-839e-058668f304c6\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " Nov 04 01:34:54.508448 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.508310 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-istio-data\") pod \"d31799de-2b0d-4f2e-839e-058668f304c6\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " Nov 04 01:34:54.508448 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.508371 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31799de-2b0d-4f2e-839e-058668f304c6-kube-api-access-p4rzv\" (UniqueName: \"kubernetes.io/projected/d31799de-2b0d-4f2e-839e-058668f304c6-kube-api-access-p4rzv\") pod \"d31799de-2b0d-4f2e-839e-058668f304c6\" (UID: \"d31799de-2b0d-4f2e-839e-058668f304c6\") " Nov 04 01:34:54.508545 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.508451 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-credential-socket" pod "d31799de-2b0d-4f2e-839e-058668f304c6" (UID: "d31799de-2b0d-4f2e-839e-058668f304c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:54.508545 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.508503 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-workload-certs" pod "d31799de-2b0d-4f2e-839e-058668f304c6" (UID: "d31799de-2b0d-4f2e-839e-058668f304c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:54.508545 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.508528 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31799de-2b0d-4f2e-839e-058668f304c6-istiod-ca-cert" pod "d31799de-2b0d-4f2e-839e-058668f304c6" (UID: "d31799de-2b0d-4f2e-839e-058668f304c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:54.508739 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.508711 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-workload-socket" pod "d31799de-2b0d-4f2e-839e-058668f304c6" (UID: "d31799de-2b0d-4f2e-839e-058668f304c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:54.508967 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.508738 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-istio-data" pod "d31799de-2b0d-4f2e-839e-058668f304c6" (UID: "d31799de-2b0d-4f2e-839e-058668f304c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:54.508967 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.508912 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:54.508967 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.508930 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:54.508967 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.508943 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:54.508967 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.508954 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:54.508967 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.508966 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31799de-2b0d-4f2e-839e-058668f304c6-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:54.510300 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.510279 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-istio-envoy" pod "d31799de-2b0d-4f2e-839e-058668f304c6" (UID: "d31799de-2b0d-4f2e-839e-058668f304c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:54.510384 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.510363 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d31799de-2b0d-4f2e-839e-058668f304c6-istio-podinfo" pod "d31799de-2b0d-4f2e-839e-058668f304c6" (UID: "d31799de-2b0d-4f2e-839e-058668f304c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:54.510958 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.510919 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31799de-2b0d-4f2e-839e-058668f304c6-istio-token" pod "d31799de-2b0d-4f2e-839e-058668f304c6" (UID: "d31799de-2b0d-4f2e-839e-058668f304c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:54.510958 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.510935 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31799de-2b0d-4f2e-839e-058668f304c6-kube-api-access-p4rzv" pod "d31799de-2b0d-4f2e-839e-058668f304c6" (UID: "d31799de-2b0d-4f2e-839e-058668f304c6"). InnerVolumeSpecName "kube-api-access-p4rzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:54.610165 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.610088 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31799de-2b0d-4f2e-839e-058668f304c6-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:54.610165 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.610113 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31799de-2b0d-4f2e-839e-058668f304c6-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:54.610165 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.610122 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31799de-2b0d-4f2e-839e-058668f304c6-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:54.610165 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:54.610131 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4rzv\" (UniqueName: \"kubernetes.io/projected/d31799de-2b0d-4f2e-839e-058668f304c6-kube-api-access-p4rzv\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:55.055999 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:55.055966 4852 generic.go:356] "Generic (PLEG): container finished" podID="d31799de-2b0d-4f2e-839e-058668f304c6" containerID="c617da0f9451141112c806b4771869ea40c51f980e5bb4b1f7b207d843c5e4f2" exitCode=0 Nov 04 01:34:55.056189 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:55.056012 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" Nov 04 01:34:55.056189 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:55.056051 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" event={"ID":"d31799de-2b0d-4f2e-839e-058668f304c6","Type":"ContainerDied","Data":"c617da0f9451141112c806b4771869ea40c51f980e5bb4b1f7b207d843c5e4f2"} Nov 04 01:34:55.056189 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:55.056115 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f" event={"ID":"d31799de-2b0d-4f2e-839e-058668f304c6","Type":"ContainerDied","Data":"c5f08847de7e1f272ed61e5b9e500e15a6e0a4c8fbdbea5252d15ec952b3d07b"} Nov 04 01:34:55.056189 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:55.056143 4852 scope.go:122] "RemoveContainer" containerID="c617da0f9451141112c806b4771869ea40c51f980e5bb4b1f7b207d843c5e4f2" Nov 04 01:34:55.066603 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:55.066587 4852 scope.go:122] "RemoveContainer" containerID="09ce9b824a363adf76899b88511527a5e5b7bed4d59032c212dafa11238668f7" Nov 04 01:34:55.075765 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:55.075734 4852 scope.go:122] "RemoveContainer" containerID="c617da0f9451141112c806b4771869ea40c51f980e5bb4b1f7b207d843c5e4f2" Nov 04 01:34:55.076098 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:34:55.076074 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c617da0f9451141112c806b4771869ea40c51f980e5bb4b1f7b207d843c5e4f2\": container with ID starting with c617da0f9451141112c806b4771869ea40c51f980e5bb4b1f7b207d843c5e4f2 not found: ID does not exist" containerID="c617da0f9451141112c806b4771869ea40c51f980e5bb4b1f7b207d843c5e4f2" Nov 04 01:34:55.076384 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:55.076109 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c617da0f9451141112c806b4771869ea40c51f980e5bb4b1f7b207d843c5e4f2"} err="failed to get container status \"c617da0f9451141112c806b4771869ea40c51f980e5bb4b1f7b207d843c5e4f2\": rpc error: code = NotFound desc = could not find container \"c617da0f9451141112c806b4771869ea40c51f980e5bb4b1f7b207d843c5e4f2\": container with ID starting with c617da0f9451141112c806b4771869ea40c51f980e5bb4b1f7b207d843c5e4f2 not found: ID does not exist" Nov 04 01:34:55.076384 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:55.076131 4852 scope.go:122] "RemoveContainer" containerID="09ce9b824a363adf76899b88511527a5e5b7bed4d59032c212dafa11238668f7" Nov 04 01:34:55.076711 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:34:55.076432 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ce9b824a363adf76899b88511527a5e5b7bed4d59032c212dafa11238668f7\": container with ID starting with 09ce9b824a363adf76899b88511527a5e5b7bed4d59032c212dafa11238668f7 not found: ID does not exist" containerID="09ce9b824a363adf76899b88511527a5e5b7bed4d59032c212dafa11238668f7" Nov 04 01:34:55.076711 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:55.076463 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ce9b824a363adf76899b88511527a5e5b7bed4d59032c212dafa11238668f7"} err="failed to get container status \"09ce9b824a363adf76899b88511527a5e5b7bed4d59032c212dafa11238668f7\": rpc error: code = NotFound desc = could not find container \"09ce9b824a363adf76899b88511527a5e5b7bed4d59032c212dafa11238668f7\": container with ID starting with 09ce9b824a363adf76899b88511527a5e5b7bed4d59032c212dafa11238668f7 not found: ID does not exist" Nov 04 01:34:55.079081 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:55.079059 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f"] Nov 04 01:34:55.083069 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:55.083049 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-af6909-openshift-default-847d464bb9-vtl8f"] Nov 04 01:34:55.121590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:55.107443 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:55.314703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:55.309752 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:55.457206 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:55.457172 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6"] Nov 04 01:34:55.477380 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:55.477348 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:55.659765 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:55.659684 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:55.866589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:55.856044 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:56.021003 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:56.020933 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:56.052590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:56.048241 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d31799de-2b0d-4f2e-839e-058668f304c6" path="/var/lib/kubelet/pods/d31799de-2b0d-4f2e-839e-058668f304c6/volumes" Nov 04 01:34:56.179641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:56.162191 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx"] Nov 04 01:34:56.179641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:56.162441 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" podUID="a5c642db-a1e8-4839-9fa7-1821588ecb15" containerName="istio-proxy" containerID="cri-o://2d69104a66c37ac0deb4b0c03e18376f90b5eb0f18c4be7182a77d2f6ee4039a" gracePeriod=30 Nov 04 01:34:56.275725 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:56.226252 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:56.314586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:56.310174 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf"] Nov 04 01:34:56.314586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:56.310425 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" podUID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" containerName="istio-proxy" containerID="cri-o://94b72b3e9f3931e8c1101a8f0969b818516cefe4ad899d9a7d6d2f46d3e78203" gracePeriod=30 Nov 04 01:34:56.359589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:56.356616 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl"] Nov 04 01:34:56.359589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:56.356848 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" podUID="9febb536-9d73-41c8-8a6e-5b2aa1446825" containerName="istio-proxy" containerID="cri-o://43f9bcd7de1208f57d06b688d5017e1b1ee3429c16c6086783527cfbf9f0cea1" gracePeriod=30 Nov 04 01:34:56.707965 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:56.707869 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:56.962685 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:56.958310 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:57.102856 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:57.102816 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:34:57.104059 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:57.104025 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" podUID="a5c642db-a1e8-4839-9fa7-1821588ecb15" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:34:57.234719 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:57.232061 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:57.819588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:57.819451 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:58.121719 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:58.121325 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:58.505216 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:58.505185 4852 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:34:58.505479 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:58.505454 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" podUID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:34:59.060590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:59.057076 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:59.606631 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:59.585971 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:34:59.780590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:34:59.780380 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:00.055617 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:35:00.053702 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:35:00.478763 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:00.478640 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6" podUID="614a961a-dc3c-41ae-963f-53dfe934eda0" containerName="payload-pre-processing" containerID="cri-o://d5908a6bcaadf7b071e3c6c45f5f43ef1f8f818254172109b92d6f3a00bdfe06" gracePeriod=25 Nov 04 01:35:00.637366 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:00.634519 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:01.241506 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:01.241457 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:35:01.247695 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:01.247671 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:01.348130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:01.347776 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6" Nov 04 01:35:01.500077 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:01.500036 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6" Nov 04 01:35:01.500751 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:01.500598 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6" event={"ID":"614a961a-dc3c-41ae-963f-53dfe934eda0","Type":"ContainerDied","Data":"d5908a6bcaadf7b071e3c6c45f5f43ef1f8f818254172109b92d6f3a00bdfe06"} Nov 04 01:35:01.500994 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:01.500958 4852 scope.go:122] "RemoveContainer" containerID="d5908a6bcaadf7b071e3c6c45f5f43ef1f8f818254172109b92d6f3a00bdfe06" Nov 04 01:35:01.513784 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:01.499949 4852 generic.go:356] "Generic (PLEG): container finished" podID="614a961a-dc3c-41ae-963f-53dfe934eda0" containerID="d5908a6bcaadf7b071e3c6c45f5f43ef1f8f818254172109b92d6f3a00bdfe06" exitCode=0 Nov 04 01:35:01.514185 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:01.514162 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6" event={"ID":"614a961a-dc3c-41ae-963f-53dfe934eda0","Type":"ContainerDied","Data":"1bd0a9cd9dea01bc785857f1367f80d7b25082385a1b8e1d29339d7ac9eb50d8"} Nov 04 01:35:01.521898 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:01.520503 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/614a961a-dc3c-41ae-963f-53dfe934eda0-kube-api-access-jhzwg\" (UniqueName: \"kubernetes.io/projected/614a961a-dc3c-41ae-963f-53dfe934eda0-kube-api-access-jhzwg\") pod \"614a961a-dc3c-41ae-963f-53dfe934eda0\" (UID: \"614a961a-dc3c-41ae-963f-53dfe934eda0\") " Nov 04 01:35:01.521898 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:01.520718 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/614a961a-dc3c-41ae-963f-53dfe934eda0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614a961a-dc3c-41ae-963f-53dfe934eda0-plugins-config-volume\") pod \"614a961a-dc3c-41ae-963f-53dfe934eda0\" (UID: \"614a961a-dc3c-41ae-963f-53dfe934eda0\") " Nov 04 01:35:01.546130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:01.535112 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/614a961a-dc3c-41ae-963f-53dfe934eda0-plugins-config-volume" pod "614a961a-dc3c-41ae-963f-53dfe934eda0" (UID: "614a961a-dc3c-41ae-963f-53dfe934eda0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:01.634680 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:01.623461 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614a961a-dc3c-41ae-963f-53dfe934eda0-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:01.682762 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:01.665603 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:01.682762 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:01.682706 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614a961a-dc3c-41ae-963f-53dfe934eda0-kube-api-access-jhzwg" pod "614a961a-dc3c-41ae-963f-53dfe934eda0" (UID: "614a961a-dc3c-41ae-963f-53dfe934eda0"). InnerVolumeSpecName "kube-api-access-jhzwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:01.753613 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:01.728419 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhzwg\" (UniqueName: \"kubernetes.io/projected/614a961a-dc3c-41ae-963f-53dfe934eda0-kube-api-access-jhzwg\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:01.793828 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:01.790743 4852 scope.go:122] "RemoveContainer" containerID="d5908a6bcaadf7b071e3c6c45f5f43ef1f8f818254172109b92d6f3a00bdfe06" Nov 04 01:35:01.845747 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:35:01.830708 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5908a6bcaadf7b071e3c6c45f5f43ef1f8f818254172109b92d6f3a00bdfe06\": container with ID starting with d5908a6bcaadf7b071e3c6c45f5f43ef1f8f818254172109b92d6f3a00bdfe06 not found: ID does not exist" containerID="d5908a6bcaadf7b071e3c6c45f5f43ef1f8f818254172109b92d6f3a00bdfe06" Nov 04 01:35:01.845747 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:01.830761 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5908a6bcaadf7b071e3c6c45f5f43ef1f8f818254172109b92d6f3a00bdfe06"} err="failed to get container status \"d5908a6bcaadf7b071e3c6c45f5f43ef1f8f818254172109b92d6f3a00bdfe06\": rpc error: code = NotFound desc = could not find container \"d5908a6bcaadf7b071e3c6c45f5f43ef1f8f818254172109b92d6f3a00bdfe06\": container with ID starting with d5908a6bcaadf7b071e3c6c45f5f43ef1f8f818254172109b92d6f3a00bdfe06 not found: ID does not exist" Nov 04 01:35:02.463909 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.463875 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6"] Nov 04 01:35:02.474112 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.474078 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ef6cc9-85c77f657f-jj7c6"] Nov 04 01:35:02.515245 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:35:02.515188 4852 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614a961a_dc3c_41ae_963f_53dfe934eda0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614a961a_dc3c_41ae_963f_53dfe934eda0.slice/crio-1bd0a9cd9dea01bc785857f1367f80d7b25082385a1b8e1d29339d7ac9eb50d8\": RecentStats: unable to find data in memory cache]" Nov 04 01:35:02.532532 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.528390 4852 generic.go:356] "Generic (PLEG): container finished" podID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" containerID="94b72b3e9f3931e8c1101a8f0969b818516cefe4ad899d9a7d6d2f46d3e78203" exitCode=0 Nov 04 01:35:02.533232 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.528447 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" event={"ID":"cae606b8-4a95-4046-a9eb-3df149dcf7a8","Type":"ContainerDied","Data":"94b72b3e9f3931e8c1101a8f0969b818516cefe4ad899d9a7d6d2f46d3e78203"} Nov 04 01:35:02.540329 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.540295 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:02.541193 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:35:02.541150 4852 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614a961a_dc3c_41ae_963f_53dfe934eda0.slice/crio-1bd0a9cd9dea01bc785857f1367f80d7b25082385a1b8e1d29339d7ac9eb50d8\": RecentStats: unable to find data in memory cache]" Nov 04 01:35:02.670217 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.670134 4852 generic.go:356] "Generic (PLEG): container finished" podID="a5c642db-a1e8-4839-9fa7-1821588ecb15" containerID="2d69104a66c37ac0deb4b0c03e18376f90b5eb0f18c4be7182a77d2f6ee4039a" exitCode=0 Nov 04 01:35:02.670654 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.670420 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" event={"ID":"a5c642db-a1e8-4839-9fa7-1821588ecb15","Type":"ContainerDied","Data":"2d69104a66c37ac0deb4b0c03e18376f90b5eb0f18c4be7182a77d2f6ee4039a"} Nov 04 01:35:02.708589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.702352 4852 generic.go:356] "Generic (PLEG): container finished" podID="9febb536-9d73-41c8-8a6e-5b2aa1446825" containerID="43f9bcd7de1208f57d06b688d5017e1b1ee3429c16c6086783527cfbf9f0cea1" exitCode=0 Nov 04 01:35:02.708589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.702411 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" event={"ID":"9febb536-9d73-41c8-8a6e-5b2aa1446825","Type":"ContainerDied","Data":"43f9bcd7de1208f57d06b688d5017e1b1ee3429c16c6086783527cfbf9f0cea1"} Nov 04 01:35:02.708589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.702480 4852 scope.go:122] "RemoveContainer" containerID="1fe10790ad7324f595f5dc7fc778f0fce0aafce8523825cab82e73850efb7e85" Nov 04 01:35:02.865266 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.865233 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:02.895790 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.895437 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:35:02.963589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.960721 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:35:03.002618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.989997 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-workload-socket\") pod \"a5c642db-a1e8-4839-9fa7-1821588ecb15\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " Nov 04 01:35:03.002618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.990126 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-credential-socket\") pod \"a5c642db-a1e8-4839-9fa7-1821588ecb15\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " Nov 04 01:35:03.002618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.990212 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-data\") pod \"a5c642db-a1e8-4839-9fa7-1821588ecb15\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " Nov 04 01:35:03.002618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.990258 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-envoy\") pod \"a5c642db-a1e8-4839-9fa7-1821588ecb15\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " Nov 04 01:35:03.002618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.990361 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c642db-a1e8-4839-9fa7-1821588ecb15-kube-api-access-mlghl\" (UniqueName: \"kubernetes.io/projected/a5c642db-a1e8-4839-9fa7-1821588ecb15-kube-api-access-mlghl\") pod \"a5c642db-a1e8-4839-9fa7-1821588ecb15\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " Nov 04 01:35:03.002618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.990414 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-workload-certs\") pod \"a5c642db-a1e8-4839-9fa7-1821588ecb15\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " Nov 04 01:35:03.002618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.990445 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-podinfo\") pod \"a5c642db-a1e8-4839-9fa7-1821588ecb15\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " Nov 04 01:35:03.002618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.990497 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5c642db-a1e8-4839-9fa7-1821588ecb15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c642db-a1e8-4839-9fa7-1821588ecb15-istiod-ca-cert\") pod \"a5c642db-a1e8-4839-9fa7-1821588ecb15\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " Nov 04 01:35:03.002618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.990543 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-token\" (UniqueName: \"kubernetes.io/projected/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-token\") pod \"a5c642db-a1e8-4839-9fa7-1821588ecb15\" (UID: \"a5c642db-a1e8-4839-9fa7-1821588ecb15\") " Nov 04 01:35:03.002618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.992843 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-credential-socket" pod "a5c642db-a1e8-4839-9fa7-1821588ecb15" (UID: "a5c642db-a1e8-4839-9fa7-1821588ecb15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:03.002618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:02.993031 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-workload-socket" pod "a5c642db-a1e8-4839-9fa7-1821588ecb15" (UID: "a5c642db-a1e8-4839-9fa7-1821588ecb15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:03.005228 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.003426 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-data" pod "a5c642db-a1e8-4839-9fa7-1821588ecb15" (UID: "a5c642db-a1e8-4839-9fa7-1821588ecb15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:03.005228 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.003814 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-workload-certs" pod "a5c642db-a1e8-4839-9fa7-1821588ecb15" (UID: "a5c642db-a1e8-4839-9fa7-1821588ecb15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:03.006840 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.005619 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c642db-a1e8-4839-9fa7-1821588ecb15-istiod-ca-cert" pod "a5c642db-a1e8-4839-9fa7-1821588ecb15" (UID: "a5c642db-a1e8-4839-9fa7-1821588ecb15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:03.127716 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.120792 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:35:03.127716 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.121459 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-token\" (UniqueName: \"kubernetes.io/projected/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-token\") pod \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " Nov 04 01:35:03.127716 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.121520 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-data\") pod \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " Nov 04 01:35:03.127716 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.121579 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-credential-socket\") pod \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " Nov 04 01:35:03.127716 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.121619 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-podinfo\") pod \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " Nov 04 01:35:03.127716 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.121701 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-workload-certs\") pod \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " Nov 04 01:35:03.127716 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.121747 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istiod-ca-cert\") pod \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " Nov 04 01:35:03.127716 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.121784 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cae606b8-4a95-4046-a9eb-3df149dcf7a8-kube-api-access-pf25q\" (UniqueName: \"kubernetes.io/projected/cae606b8-4a95-4046-a9eb-3df149dcf7a8-kube-api-access-pf25q\") pod \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " Nov 04 01:35:03.127716 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.121834 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-workload-socket\") pod \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " Nov 04 01:35:03.127716 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.121922 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-envoy\") pod \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\" (UID: \"cae606b8-4a95-4046-a9eb-3df149dcf7a8\") " Nov 04 01:35:03.168594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.137806 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-data" pod "cae606b8-4a95-4046-a9eb-3df149dcf7a8" (UID: "cae606b8-4a95-4046-a9eb-3df149dcf7a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:03.168594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.138014 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-credential-socket" pod "cae606b8-4a95-4046-a9eb-3df149dcf7a8" (UID: "cae606b8-4a95-4046-a9eb-3df149dcf7a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:03.168594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.141753 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istiod-ca-cert" pod "cae606b8-4a95-4046-a9eb-3df149dcf7a8" (UID: "cae606b8-4a95-4046-a9eb-3df149dcf7a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:03.168594 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.153641 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-workload-certs" pod "cae606b8-4a95-4046-a9eb-3df149dcf7a8" (UID: "cae606b8-4a95-4046-a9eb-3df149dcf7a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:03.172684 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.172640 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-workload-socket" pod "cae606b8-4a95-4046-a9eb-3df149dcf7a8" (UID: "cae606b8-4a95-4046-a9eb-3df149dcf7a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:03.185592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.183119 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.185592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.183152 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.185592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.183171 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c642db-a1e8-4839-9fa7-1821588ecb15-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.185592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.183187 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.185592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.183201 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.185592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.183217 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.185592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.183231 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.185592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.183244 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.185592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.183258 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.185592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.183272 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.185592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.183991 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-podinfo" pod "cae606b8-4a95-4046-a9eb-3df149dcf7a8" (UID: "cae606b8-4a95-4046-a9eb-3df149dcf7a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:03.191306 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.191263 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-token" pod "cae606b8-4a95-4046-a9eb-3df149dcf7a8" (UID: "cae606b8-4a95-4046-a9eb-3df149dcf7a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:03.198048 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.198007 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae606b8-4a95-4046-a9eb-3df149dcf7a8-kube-api-access-pf25q" pod "cae606b8-4a95-4046-a9eb-3df149dcf7a8" (UID: "cae606b8-4a95-4046-a9eb-3df149dcf7a8"). InnerVolumeSpecName "kube-api-access-pf25q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:03.248601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.236064 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:03.295598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.284437 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-podinfo\") pod \"9febb536-9d73-41c8-8a6e-5b2aa1446825\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " Nov 04 01:35:03.295598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.284504 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9febb536-9d73-41c8-8a6e-5b2aa1446825-kube-api-access-mhqw7\" (UniqueName: \"kubernetes.io/projected/9febb536-9d73-41c8-8a6e-5b2aa1446825-kube-api-access-mhqw7\") pod \"9febb536-9d73-41c8-8a6e-5b2aa1446825\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " Nov 04 01:35:03.295598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.284545 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-workload-certs\") pod \"9febb536-9d73-41c8-8a6e-5b2aa1446825\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " Nov 04 01:35:03.295598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.284611 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-workload-socket\") pod \"9febb536-9d73-41c8-8a6e-5b2aa1446825\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " Nov 04 01:35:03.295598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.284672 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-envoy\") pod \"9febb536-9d73-41c8-8a6e-5b2aa1446825\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " Nov 04 01:35:03.295598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.284709 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-credential-socket\") pod \"9febb536-9d73-41c8-8a6e-5b2aa1446825\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " Nov 04 01:35:03.295598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.284750 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9febb536-9d73-41c8-8a6e-5b2aa1446825-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9febb536-9d73-41c8-8a6e-5b2aa1446825-istiod-ca-cert\") pod \"9febb536-9d73-41c8-8a6e-5b2aa1446825\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " Nov 04 01:35:03.295598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.284783 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-token\" (UniqueName: \"kubernetes.io/projected/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-token\") pod \"9febb536-9d73-41c8-8a6e-5b2aa1446825\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " Nov 04 01:35:03.295598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.284825 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-data\") pod \"9febb536-9d73-41c8-8a6e-5b2aa1446825\" (UID: \"9febb536-9d73-41c8-8a6e-5b2aa1446825\") " Nov 04 01:35:03.295598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.285286 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.295598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.285306 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.295598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.285323 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf25q\" (UniqueName: \"kubernetes.io/projected/cae606b8-4a95-4046-a9eb-3df149dcf7a8-kube-api-access-pf25q\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.295598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.286822 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-workload-socket" pod "9febb536-9d73-41c8-8a6e-5b2aa1446825" (UID: "9febb536-9d73-41c8-8a6e-5b2aa1446825"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:03.323588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.309124 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-envoy" pod "cae606b8-4a95-4046-a9eb-3df149dcf7a8" (UID: "cae606b8-4a95-4046-a9eb-3df149dcf7a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:03.323588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.309396 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-workload-certs" pod "9febb536-9d73-41c8-8a6e-5b2aa1446825" (UID: "9febb536-9d73-41c8-8a6e-5b2aa1446825"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:03.323588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.315758 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9febb536-9d73-41c8-8a6e-5b2aa1446825-istiod-ca-cert" pod "9febb536-9d73-41c8-8a6e-5b2aa1446825" (UID: "9febb536-9d73-41c8-8a6e-5b2aa1446825"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:03.324104 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.324077 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-data" pod "9febb536-9d73-41c8-8a6e-5b2aa1446825" (UID: "9febb536-9d73-41c8-8a6e-5b2aa1446825"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:03.352797 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.327663 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-credential-socket" pod "9febb536-9d73-41c8-8a6e-5b2aa1446825" (UID: "9febb536-9d73-41c8-8a6e-5b2aa1446825"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:03.355781 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.355737 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9febb536-9d73-41c8-8a6e-5b2aa1446825-kube-api-access-mhqw7" pod "9febb536-9d73-41c8-8a6e-5b2aa1446825" (UID: "9febb536-9d73-41c8-8a6e-5b2aa1446825"). InnerVolumeSpecName "kube-api-access-mhqw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:03.417544 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.417451 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.444842 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.444816 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae606b8-4a95-4046-a9eb-3df149dcf7a8-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.445042 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.445026 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.445197 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.445181 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9febb536-9d73-41c8-8a6e-5b2aa1446825-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.445305 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.445292 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.445408 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.445395 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhqw7\" (UniqueName: \"kubernetes.io/projected/9febb536-9d73-41c8-8a6e-5b2aa1446825-kube-api-access-mhqw7\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.445502 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.445490 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.509621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.509591 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:03.517605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.514695 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-envoy" pod "9febb536-9d73-41c8-8a6e-5b2aa1446825" (UID: "9febb536-9d73-41c8-8a6e-5b2aa1446825"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:03.521970 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.521928 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-podinfo" pod "9febb536-9d73-41c8-8a6e-5b2aa1446825" (UID: "9febb536-9d73-41c8-8a6e-5b2aa1446825"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:03.522214 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.522194 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-token" pod "9febb536-9d73-41c8-8a6e-5b2aa1446825" (UID: "9febb536-9d73-41c8-8a6e-5b2aa1446825"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:03.568448 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.568365 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.568448 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.568398 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.568448 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.568413 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9febb536-9d73-41c8-8a6e-5b2aa1446825-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.666796 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.664919 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-podinfo" pod "a5c642db-a1e8-4839-9fa7-1821588ecb15" (UID: "a5c642db-a1e8-4839-9fa7-1821588ecb15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:03.666796 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.665092 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-token" pod "a5c642db-a1e8-4839-9fa7-1821588ecb15" (UID: "a5c642db-a1e8-4839-9fa7-1821588ecb15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:03.666796 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.666690 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-envoy" pod "a5c642db-a1e8-4839-9fa7-1821588ecb15" (UID: "a5c642db-a1e8-4839-9fa7-1821588ecb15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:03.673625 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.670308 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.673625 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.670340 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.673625 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.670356 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c642db-a1e8-4839-9fa7-1821588ecb15-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.701762 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.701693 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c642db-a1e8-4839-9fa7-1821588ecb15-kube-api-access-mlghl" pod "a5c642db-a1e8-4839-9fa7-1821588ecb15" (UID: "a5c642db-a1e8-4839-9fa7-1821588ecb15"). InnerVolumeSpecName "kube-api-access-mlghl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:03.721735 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.719431 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" event={"ID":"cae606b8-4a95-4046-a9eb-3df149dcf7a8","Type":"ContainerDied","Data":"31090203615dd32de09b4623a0cbfab2f65bc0a41f4816548b25cda7719c6432"} Nov 04 01:35:03.721735 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.719503 4852 scope.go:122] "RemoveContainer" containerID="94b72b3e9f3931e8c1101a8f0969b818516cefe4ad899d9a7d6d2f46d3e78203" Nov 04 01:35:03.721735 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.719643 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf" Nov 04 01:35:03.733394 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.728363 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" event={"ID":"a5c642db-a1e8-4839-9fa7-1821588ecb15","Type":"ContainerDied","Data":"5ab4e11a171abb7a1b2c91c0fb4eb011e54bf46af1152bc1051b69f234b3fa65"} Nov 04 01:35:03.733394 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.728470 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx" Nov 04 01:35:03.738635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.734738 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" event={"ID":"9febb536-9d73-41c8-8a6e-5b2aa1446825","Type":"ContainerDied","Data":"6c310d2d5cd9c10dbc64e197a0a00afce52faebe048fc593dc5be04a068ecedc"} Nov 04 01:35:03.738635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.734832 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl" Nov 04 01:35:03.772008 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.771971 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlghl\" (UniqueName: \"kubernetes.io/projected/a5c642db-a1e8-4839-9fa7-1821588ecb15-kube-api-access-mlghl\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:03.796305 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.796284 4852 scope.go:122] "RemoveContainer" containerID="2d69104a66c37ac0deb4b0c03e18376f90b5eb0f18c4be7182a77d2f6ee4039a" Nov 04 01:35:03.810889 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.810839 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx"] Nov 04 01:35:03.836994 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.836199 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vg4vx"] Nov 04 01:35:03.837898 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.837879 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf"] Nov 04 01:35:03.846587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.841680 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-52dmf"] Nov 04 01:35:03.854606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.851836 4852 scope.go:122] "RemoveContainer" containerID="43f9bcd7de1208f57d06b688d5017e1b1ee3429c16c6086783527cfbf9f0cea1" Nov 04 01:35:03.860934 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.860918 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl"] Nov 04 01:35:03.867454 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:03.867434 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dfcebf-openshift-default-68c6c5cb87-5qmpl"] Nov 04 01:35:04.003638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:04.003609 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="614a961a-dc3c-41ae-963f-53dfe934eda0" path="/var/lib/kubelet/pods/614a961a-dc3c-41ae-963f-53dfe934eda0/volumes" Nov 04 01:35:04.004513 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:04.004491 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9febb536-9d73-41c8-8a6e-5b2aa1446825" path="/var/lib/kubelet/pods/9febb536-9d73-41c8-8a6e-5b2aa1446825/volumes" Nov 04 01:35:04.005300 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:04.005269 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5c642db-a1e8-4839-9fa7-1821588ecb15" path="/var/lib/kubelet/pods/a5c642db-a1e8-4839-9fa7-1821588ecb15/volumes" Nov 04 01:35:04.006496 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:04.006477 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" path="/var/lib/kubelet/pods/cae606b8-4a95-4046-a9eb-3df149dcf7a8/volumes" Nov 04 01:35:04.150100 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:04.148647 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:04.167119 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:04.166996 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-tqg84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:35:04.167119 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:04.167048 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" podUID="c13769b4-f82d-460e-aa0f-0e882b40346c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:04.948277 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:04.948175 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:05.762871 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:05.762831 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4"] Nov 04 01:35:05.803508 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:05.803470 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4"] Nov 04 01:35:05.921058 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:05.920980 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:06.125454 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:06.125344 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:06.155593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:06.154669 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:35:06.155593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:06.154745 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:06.878018 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:06.877951 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:07.131872 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:07.131781 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:35:07.132349 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:07.131868 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:07.196813 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:07.196768 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:07.434638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:07.430678 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" containerID="cri-o://8b4cd5bc5d755efafac4fc022f79f3276b2f3335e22191760186e9e2359e1f52" gracePeriod=45 Nov 04 01:35:07.434638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:07.430839 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" containerID="cri-o://c09853f05e48f4f115111700306daa592e7d322ddb4eebf9077cdb95b26c8adc" gracePeriod=45 Nov 04 01:35:07.570165 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:07.570132 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:07.778539 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:07.778347 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:07.783908 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:07.783744 4852 generic.go:356] "Generic (PLEG): container finished" podID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerID="c09853f05e48f4f115111700306daa592e7d322ddb4eebf9077cdb95b26c8adc" exitCode=0 Nov 04 01:35:07.783908 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:07.783843 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" event={"ID":"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d","Type":"ContainerDied","Data":"c09853f05e48f4f115111700306daa592e7d322ddb4eebf9077cdb95b26c8adc"} Nov 04 01:35:07.786286 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:07.786263 4852 generic.go:356] "Generic (PLEG): container finished" podID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerID="8b4cd5bc5d755efafac4fc022f79f3276b2f3335e22191760186e9e2359e1f52" exitCode=0 Nov 04 01:35:07.788001 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:07.787977 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" event={"ID":"c8ff057d-6968-4e4a-b0cf-3c5fbc325173","Type":"ContainerDied","Data":"8b4cd5bc5d755efafac4fc022f79f3276b2f3335e22191760186e9e2359e1f52"} Nov 04 01:35:07.890505 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:07.890253 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:35:07.957924 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:07.957597 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:35:07.975493 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:07.975448 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv"] Nov 04 01:35:07.976521 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:07.976144 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" containerID="cri-o://0d64beb5917027fc312d6eaa92b4ed17095674258edb6ecbdad6b50843d2dba0" gracePeriod=30 Nov 04 01:35:07.985617 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:07.983623 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn"] Nov 04 01:35:07.985617 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:07.983848 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerName="istio-proxy" containerID="cri-o://4d3e9f4714237102b221f4ce40a934c4eebe7fa18ca6ef148f5fe30b0aed4a28" gracePeriod=30 Nov 04 01:35:08.009892 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.009863 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x"] Nov 04 01:35:08.010123 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.010095 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" containerID="cri-o://b35f53e833e0d72738cf6e6f08c7c522edbd7f720f752b37886e33581bcbfce7" gracePeriod=30 Nov 04 01:35:08.086027 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.085994 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:08.087049 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.087020 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-home\" (UniqueName: \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-home\") pod \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " Nov 04 01:35:08.087178 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.087066 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-model-cache\") pod \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " Nov 04 01:35:08.087178 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.087134 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-tls-certs\" (UniqueName: \"kubernetes.io/secret/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-tls-certs\") pod \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " Nov 04 01:35:08.087311 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.087177 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-model-cache\") pod \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " Nov 04 01:35:08.087311 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.087223 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-kube-api-access-4g9w6\" (UniqueName: \"kubernetes.io/projected/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-kube-api-access-4g9w6\") pod \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " Nov 04 01:35:08.087311 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.087264 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-dshm\") pod \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " Nov 04 01:35:08.087311 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.087263 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-home" pod "c8ff057d-6968-4e4a-b0cf-3c5fbc325173" (UID: "c8ff057d-6968-4e4a-b0cf-3c5fbc325173"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:08.087541 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.087331 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-home\" (UniqueName: \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-home\") pod \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " Nov 04 01:35:08.087541 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.087377 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-tmp-dir\") pod \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " Nov 04 01:35:08.087541 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.087366 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-model-cache" pod "c8ff057d-6968-4e4a-b0cf-3c5fbc325173" (UID: "c8ff057d-6968-4e4a-b0cf-3c5fbc325173"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:08.087541 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.087425 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-tmp-dir\") pod \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " Nov 04 01:35:08.087541 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.087439 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-model-cache" pod "7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" (UID: "7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:08.087541 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.087461 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-dshm\") pod \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\" (UID: \"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d\") " Nov 04 01:35:08.087905 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.087585 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-tls-certs\" (UniqueName: \"kubernetes.io/secret/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-tls-certs\") pod \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " Nov 04 01:35:08.087905 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.087615 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-kube-api-access-l8wcq\" (UniqueName: \"kubernetes.io/projected/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-kube-api-access-l8wcq\") pod \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\" (UID: \"c8ff057d-6968-4e4a-b0cf-3c5fbc325173\") " Nov 04 01:35:08.087905 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.087662 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-tmp-dir" pod "7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" (UID: "7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:08.087905 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.087783 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-home" pod "7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" (UID: "7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:08.088116 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.087958 4852 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-model-cache\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.088116 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.087969 4852 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-home\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.088116 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.087978 4852 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-tmp-dir\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.088116 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.087985 4852 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-home\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.088116 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.087993 4852 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-model-cache\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.088582 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.088548 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-tmp-dir" pod "c8ff057d-6968-4e4a-b0cf-3c5fbc325173" (UID: "c8ff057d-6968-4e4a-b0cf-3c5fbc325173"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:08.090761 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.090729 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-dshm" pod "c8ff057d-6968-4e4a-b0cf-3c5fbc325173" (UID: "c8ff057d-6968-4e4a-b0cf-3c5fbc325173"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:08.090965 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.090945 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-tls-certs" pod "c8ff057d-6968-4e4a-b0cf-3c5fbc325173" (UID: "c8ff057d-6968-4e4a-b0cf-3c5fbc325173"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:08.091040 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.090978 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-kube-api-access-l8wcq" pod "c8ff057d-6968-4e4a-b0cf-3c5fbc325173" (UID: "c8ff057d-6968-4e4a-b0cf-3c5fbc325173"). InnerVolumeSpecName "kube-api-access-l8wcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:08.092199 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.092174 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-dshm" pod "7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" (UID: "7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:08.092756 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.092733 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-kube-api-access-4g9w6" pod "7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" (UID: "7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d"). InnerVolumeSpecName "kube-api-access-4g9w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:08.092922 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.092900 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-tls-certs" pod "7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" (UID: "7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:08.194383 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.192991 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g9w6\" (UniqueName: \"kubernetes.io/projected/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-kube-api-access-4g9w6\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.194383 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.193020 4852 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-dshm\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.194383 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.193030 4852 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-tmp-dir\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.194383 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.193038 4852 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-dshm\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.194383 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.193046 4852 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-tls-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.194383 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.193056 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8wcq\" (UniqueName: \"kubernetes.io/projected/c8ff057d-6968-4e4a-b0cf-3c5fbc325173-kube-api-access-l8wcq\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.194383 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.193069 4852 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d-tls-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.266297 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.266205 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:08.547449 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.547408 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:08.798517 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.798421 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" event={"ID":"7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d","Type":"ContainerDied","Data":"6dd782dd2d2f79ab1c5b9e8e08cf1c746536932194fc7582d5a1a6c6fc137346"} Nov 04 01:35:08.798517 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.798483 4852 scope.go:122] "RemoveContainer" containerID="c09853f05e48f4f115111700306daa592e7d322ddb4eebf9077cdb95b26c8adc" Nov 04 01:35:08.798795 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.798660 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx" Nov 04 01:35:08.801253 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.801220 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" event={"ID":"c8ff057d-6968-4e4a-b0cf-3c5fbc325173","Type":"ContainerDied","Data":"2c42bf48c51b8e0d3fe1b9224ec38682398fbc88e3c4767ed4ec7cce2a957a93"} Nov 04 01:35:08.801421 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.801318 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94" Nov 04 01:35:08.813854 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.813781 4852 scope.go:122] "RemoveContainer" containerID="8b4cd5bc5d755efafac4fc022f79f3276b2f3335e22191760186e9e2359e1f52" Nov 04 01:35:08.832539 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.832514 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx"] Nov 04 01:35:08.837819 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.837794 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/premium-w3-ca55c1-kserve-79dd86bcc-wx5qx"] Nov 04 01:35:08.859468 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.859445 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94"] Nov 04 01:35:08.865863 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:08.865841 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/simulator-w3-ca55c1-kserve-588fc66fb-6ql94"] Nov 04 01:35:09.260073 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.259076 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:09.347342 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.347249 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:35:09.347342 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.347301 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:09.424050 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.424025 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:35:09.427643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.427623 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:35:09.515225 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.515190 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istiod-ca-cert\") pod \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " Nov 04 01:35:09.515340 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.515234 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-workload-socket\") pod \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " Nov 04 01:35:09.515340 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.515268 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-envoy\") pod \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " Nov 04 01:35:09.515340 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.515312 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-token\" (UniqueName: \"kubernetes.io/projected/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-token\") pod \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " Nov 04 01:35:09.515500 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.515339 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-workload-certs\") pod \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " Nov 04 01:35:09.515500 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.515368 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-kube-api-access-hqrnr\" (UniqueName: \"kubernetes.io/projected/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-kube-api-access-hqrnr\") pod \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " Nov 04 01:35:09.515651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.515586 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istiod-ca-cert" pod "71f0b9b3-5fd3-4857-967f-a18a6fb481d7" (UID: "71f0b9b3-5fd3-4857-967f-a18a6fb481d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:09.516154 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.516127 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-workload-certs" pod "f84bf0b4-1021-4c15-8ce0-e7b77b11f850" (UID: "f84bf0b4-1021-4c15-8ce0-e7b77b11f850"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.516301 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.516286 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-token\" (UniqueName: \"kubernetes.io/projected/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-token\") pod \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " Nov 04 01:35:09.516381 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.516370 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-data\") pod \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " Nov 04 01:35:09.516443 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.516413 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-workload-certs\") pod \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " Nov 04 01:35:09.516504 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.516446 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-kube-api-access-j5mcw\" (UniqueName: \"kubernetes.io/projected/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-kube-api-access-j5mcw\") pod \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " Nov 04 01:35:09.516504 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.516486 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-workload-socket\") pod \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " Nov 04 01:35:09.516657 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.516527 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-podinfo\") pod \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " Nov 04 01:35:09.516777 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.516690 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istiod-ca-cert\") pod \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " Nov 04 01:35:09.516844 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.516781 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-data\") pod \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " Nov 04 01:35:09.516844 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.516819 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-envoy\") pod \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " Nov 04 01:35:09.516945 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.516866 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-credential-socket\") pod \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " Nov 04 01:35:09.516945 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.516900 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-credential-socket\") pod \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\" (UID: \"71f0b9b3-5fd3-4857-967f-a18a6fb481d7\") " Nov 04 01:35:09.517041 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.516964 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-podinfo\") pod \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\" (UID: \"f84bf0b4-1021-4c15-8ce0-e7b77b11f850\") " Nov 04 01:35:09.517435 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.517415 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.517513 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.517443 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.518377 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.518235 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-workload-socket" pod "71f0b9b3-5fd3-4857-967f-a18a6fb481d7" (UID: "71f0b9b3-5fd3-4857-967f-a18a6fb481d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.518506 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.518459 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istiod-ca-cert" pod "f84bf0b4-1021-4c15-8ce0-e7b77b11f850" (UID: "f84bf0b4-1021-4c15-8ce0-e7b77b11f850"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:09.518925 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.518750 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-credential-socket" pod "f84bf0b4-1021-4c15-8ce0-e7b77b11f850" (UID: "f84bf0b4-1021-4c15-8ce0-e7b77b11f850"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.518925 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.518755 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-workload-certs" pod "71f0b9b3-5fd3-4857-967f-a18a6fb481d7" (UID: "71f0b9b3-5fd3-4857-967f-a18a6fb481d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.519084 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.518950 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-credential-socket" pod "71f0b9b3-5fd3-4857-967f-a18a6fb481d7" (UID: "71f0b9b3-5fd3-4857-967f-a18a6fb481d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.519151 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.519116 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-data" pod "f84bf0b4-1021-4c15-8ce0-e7b77b11f850" (UID: "f84bf0b4-1021-4c15-8ce0-e7b77b11f850"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.519436 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.519268 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-data" pod "71f0b9b3-5fd3-4857-967f-a18a6fb481d7" (UID: "71f0b9b3-5fd3-4857-967f-a18a6fb481d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.519616 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.519584 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-workload-socket" pod "f84bf0b4-1021-4c15-8ce0-e7b77b11f850" (UID: "f84bf0b4-1021-4c15-8ce0-e7b77b11f850"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.520554 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.520519 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-podinfo" pod "f84bf0b4-1021-4c15-8ce0-e7b77b11f850" (UID: "f84bf0b4-1021-4c15-8ce0-e7b77b11f850"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:09.521198 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.520899 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-envoy" pod "71f0b9b3-5fd3-4857-967f-a18a6fb481d7" (UID: "71f0b9b3-5fd3-4857-967f-a18a6fb481d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.521198 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.521162 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-envoy" pod "f84bf0b4-1021-4c15-8ce0-e7b77b11f850" (UID: "f84bf0b4-1021-4c15-8ce0-e7b77b11f850"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.524919 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.524899 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-podinfo" pod "71f0b9b3-5fd3-4857-967f-a18a6fb481d7" (UID: "71f0b9b3-5fd3-4857-967f-a18a6fb481d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:09.525022 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.524934 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-token" pod "71f0b9b3-5fd3-4857-967f-a18a6fb481d7" (UID: "71f0b9b3-5fd3-4857-967f-a18a6fb481d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:09.525022 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.524999 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-kube-api-access-hqrnr" pod "71f0b9b3-5fd3-4857-967f-a18a6fb481d7" (UID: "71f0b9b3-5fd3-4857-967f-a18a6fb481d7"). InnerVolumeSpecName "kube-api-access-hqrnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:09.525682 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.525066 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-token" pod "f84bf0b4-1021-4c15-8ce0-e7b77b11f850" (UID: "f84bf0b4-1021-4c15-8ce0-e7b77b11f850"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:09.526667 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.526648 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-kube-api-access-j5mcw" pod "f84bf0b4-1021-4c15-8ce0-e7b77b11f850" (UID: "f84bf0b4-1021-4c15-8ce0-e7b77b11f850"). InnerVolumeSpecName "kube-api-access-j5mcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:09.583810 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.583789 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:35:09.618137 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.618113 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-workload-socket\") pod \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " Nov 04 01:35:09.618271 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.618156 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istiod-ca-cert\") pod \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " Nov 04 01:35:09.618271 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.618190 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-data\") pod \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " Nov 04 01:35:09.618271 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.618216 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-credential-socket\") pod \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " Nov 04 01:35:09.618271 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.618252 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-workload-certs\") pod \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " Nov 04 01:35:09.618497 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.618290 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-kube-api-access-6hsjs\" (UniqueName: \"kubernetes.io/projected/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-kube-api-access-6hsjs\") pod \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " Nov 04 01:35:09.618497 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.618370 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-token\" (UniqueName: \"kubernetes.io/projected/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-token\") pod \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " Nov 04 01:35:09.618497 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.618381 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-workload-socket" pod "5f385f89-e8a1-42b1-a6e1-839aaf1c3530" (UID: "5f385f89-e8a1-42b1-a6e1-839aaf1c3530"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.618497 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.618444 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-envoy\") pod \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " Nov 04 01:35:09.618497 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.618486 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-data" pod "5f385f89-e8a1-42b1-a6e1-839aaf1c3530" (UID: "5f385f89-e8a1-42b1-a6e1-839aaf1c3530"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.618768 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.618531 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-podinfo\") pod \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\" (UID: \"5f385f89-e8a1-42b1-a6e1-839aaf1c3530\") " Nov 04 01:35:09.618768 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.618614 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-workload-certs" pod "5f385f89-e8a1-42b1-a6e1-839aaf1c3530" (UID: "5f385f89-e8a1-42b1-a6e1-839aaf1c3530"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.618768 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.618662 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istiod-ca-cert" pod "5f385f89-e8a1-42b1-a6e1-839aaf1c3530" (UID: "5f385f89-e8a1-42b1-a6e1-839aaf1c3530"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:09.618922 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.618813 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-credential-socket" pod "5f385f89-e8a1-42b1-a6e1-839aaf1c3530" (UID: "5f385f89-e8a1-42b1-a6e1-839aaf1c3530"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.619011 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.618992 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.619078 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.619018 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.619078 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.619032 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqrnr\" (UniqueName: \"kubernetes.io/projected/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-kube-api-access-hqrnr\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.619078 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.619047 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.619078 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.619060 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.619078 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.619072 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.619346 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.619085 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5mcw\" (UniqueName: \"kubernetes.io/projected/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-kube-api-access-j5mcw\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.619346 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.619098 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.619346 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.619111 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.619346 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.619125 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.619346 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.619139 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.619346 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.619151 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.619346 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.619166 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.619346 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.619180 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.619346 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.619204 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.619346 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.619317 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.619346 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.619379 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.619346 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.619393 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.619346 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.619406 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.620259 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.619419 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84bf0b4-1021-4c15-8ce0-e7b77b11f850-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.620259 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.619433 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f0b9b3-5fd3-4857-967f-a18a6fb481d7-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.621177 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.621141 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-kube-api-access-6hsjs" pod "5f385f89-e8a1-42b1-a6e1-839aaf1c3530" (UID: "5f385f89-e8a1-42b1-a6e1-839aaf1c3530"). InnerVolumeSpecName "kube-api-access-6hsjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:09.621281 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.621236 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-envoy" pod "5f385f89-e8a1-42b1-a6e1-839aaf1c3530" (UID: "5f385f89-e8a1-42b1-a6e1-839aaf1c3530"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.621281 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.621266 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-token" pod "5f385f89-e8a1-42b1-a6e1-839aaf1c3530" (UID: "5f385f89-e8a1-42b1-a6e1-839aaf1c3530"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:09.621418 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.621397 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-podinfo" pod "5f385f89-e8a1-42b1-a6e1-839aaf1c3530" (UID: "5f385f89-e8a1-42b1-a6e1-839aaf1c3530"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:09.696110 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.696050 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:09.727877 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.721236 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hsjs\" (UniqueName: \"kubernetes.io/projected/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-kube-api-access-6hsjs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.727877 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.721299 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.727877 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.721372 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.727877 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.721408 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f385f89-e8a1-42b1-a6e1-839aaf1c3530-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.808554 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.808528 4852 generic.go:356] "Generic (PLEG): container finished" podID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerID="0d64beb5917027fc312d6eaa92b4ed17095674258edb6ecbdad6b50843d2dba0" exitCode=0 Nov 04 01:35:09.808683 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.808597 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" Nov 04 01:35:09.808683 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.808607 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" event={"ID":"71f0b9b3-5fd3-4857-967f-a18a6fb481d7","Type":"ContainerDied","Data":"0d64beb5917027fc312d6eaa92b4ed17095674258edb6ecbdad6b50843d2dba0"} Nov 04 01:35:09.808683 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.808681 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv" event={"ID":"71f0b9b3-5fd3-4857-967f-a18a6fb481d7","Type":"ContainerDied","Data":"6c8a03ef643c4496a6ffd7bf1f7551f7883d81facc25d8f41dbb0645840211a9"} Nov 04 01:35:09.808839 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.808706 4852 scope.go:122] "RemoveContainer" containerID="0d64beb5917027fc312d6eaa92b4ed17095674258edb6ecbdad6b50843d2dba0" Nov 04 01:35:09.810275 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.810250 4852 generic.go:356] "Generic (PLEG): container finished" podID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerID="b35f53e833e0d72738cf6e6f08c7c522edbd7f720f752b37886e33581bcbfce7" exitCode=0 Nov 04 01:35:09.810385 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.810313 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" Nov 04 01:35:09.810455 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.810304 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" event={"ID":"5f385f89-e8a1-42b1-a6e1-839aaf1c3530","Type":"ContainerDied","Data":"b35f53e833e0d72738cf6e6f08c7c522edbd7f720f752b37886e33581bcbfce7"} Nov 04 01:35:09.810516 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.810458 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x" event={"ID":"5f385f89-e8a1-42b1-a6e1-839aaf1c3530","Type":"ContainerDied","Data":"4ca3a8ba2f14659d48ff2af473731c7af65db835d4a1c4a25dfe0f4f7edfbd16"} Nov 04 01:35:09.815757 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.812773 4852 generic.go:356] "Generic (PLEG): container finished" podID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerID="4d3e9f4714237102b221f4ce40a934c4eebe7fa18ca6ef148f5fe30b0aed4a28" exitCode=0 Nov 04 01:35:09.815757 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.812890 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" event={"ID":"f84bf0b4-1021-4c15-8ce0-e7b77b11f850","Type":"ContainerDied","Data":"4d3e9f4714237102b221f4ce40a934c4eebe7fa18ca6ef148f5fe30b0aed4a28"} Nov 04 01:35:09.815757 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.812923 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" event={"ID":"f84bf0b4-1021-4c15-8ce0-e7b77b11f850","Type":"ContainerDied","Data":"258f8635d42971be0f31ac465be636768e3d92a2885b16aa5171da748651821e"} Nov 04 01:35:09.815757 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.812996 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn" Nov 04 01:35:09.826446 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.826418 4852 scope.go:122] "RemoveContainer" containerID="0d64beb5917027fc312d6eaa92b4ed17095674258edb6ecbdad6b50843d2dba0" Nov 04 01:35:09.826759 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:35:09.826739 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d64beb5917027fc312d6eaa92b4ed17095674258edb6ecbdad6b50843d2dba0\": container with ID starting with 0d64beb5917027fc312d6eaa92b4ed17095674258edb6ecbdad6b50843d2dba0 not found: ID does not exist" containerID="0d64beb5917027fc312d6eaa92b4ed17095674258edb6ecbdad6b50843d2dba0" Nov 04 01:35:09.826841 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.826764 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d64beb5917027fc312d6eaa92b4ed17095674258edb6ecbdad6b50843d2dba0"} err="failed to get container status \"0d64beb5917027fc312d6eaa92b4ed17095674258edb6ecbdad6b50843d2dba0\": rpc error: code = NotFound desc = could not find container \"0d64beb5917027fc312d6eaa92b4ed17095674258edb6ecbdad6b50843d2dba0\": container with ID starting with 0d64beb5917027fc312d6eaa92b4ed17095674258edb6ecbdad6b50843d2dba0 not found: ID does not exist" Nov 04 01:35:09.826841 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.826780 4852 scope.go:122] "RemoveContainer" containerID="b35f53e833e0d72738cf6e6f08c7c522edbd7f720f752b37886e33581bcbfce7" Nov 04 01:35:09.835003 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.834990 4852 scope.go:122] "RemoveContainer" containerID="c24a651f23c86541edda8f10c95a3b3f90937f808bfaa308d9ed393b96dd271d" Nov 04 01:35:09.842833 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.842817 4852 scope.go:122] "RemoveContainer" containerID="b35f53e833e0d72738cf6e6f08c7c522edbd7f720f752b37886e33581bcbfce7" Nov 04 01:35:09.843113 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:35:09.843093 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b35f53e833e0d72738cf6e6f08c7c522edbd7f720f752b37886e33581bcbfce7\": container with ID starting with b35f53e833e0d72738cf6e6f08c7c522edbd7f720f752b37886e33581bcbfce7 not found: ID does not exist" containerID="b35f53e833e0d72738cf6e6f08c7c522edbd7f720f752b37886e33581bcbfce7" Nov 04 01:35:09.843189 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.843118 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b35f53e833e0d72738cf6e6f08c7c522edbd7f720f752b37886e33581bcbfce7"} err="failed to get container status \"b35f53e833e0d72738cf6e6f08c7c522edbd7f720f752b37886e33581bcbfce7\": rpc error: code = NotFound desc = could not find container \"b35f53e833e0d72738cf6e6f08c7c522edbd7f720f752b37886e33581bcbfce7\": container with ID starting with b35f53e833e0d72738cf6e6f08c7c522edbd7f720f752b37886e33581bcbfce7 not found: ID does not exist" Nov 04 01:35:09.843189 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.843135 4852 scope.go:122] "RemoveContainer" containerID="c24a651f23c86541edda8f10c95a3b3f90937f808bfaa308d9ed393b96dd271d" Nov 04 01:35:09.843426 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:35:09.843411 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c24a651f23c86541edda8f10c95a3b3f90937f808bfaa308d9ed393b96dd271d\": container with ID starting with c24a651f23c86541edda8f10c95a3b3f90937f808bfaa308d9ed393b96dd271d not found: ID does not exist" containerID="c24a651f23c86541edda8f10c95a3b3f90937f808bfaa308d9ed393b96dd271d" Nov 04 01:35:09.843482 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.843432 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c24a651f23c86541edda8f10c95a3b3f90937f808bfaa308d9ed393b96dd271d"} err="failed to get container status \"c24a651f23c86541edda8f10c95a3b3f90937f808bfaa308d9ed393b96dd271d\": rpc error: code = NotFound desc = could not find container \"c24a651f23c86541edda8f10c95a3b3f90937f808bfaa308d9ed393b96dd271d\": container with ID starting with c24a651f23c86541edda8f10c95a3b3f90937f808bfaa308d9ed393b96dd271d not found: ID does not exist" Nov 04 01:35:09.843482 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.843446 4852 scope.go:122] "RemoveContainer" containerID="4d3e9f4714237102b221f4ce40a934c4eebe7fa18ca6ef148f5fe30b0aed4a28" Nov 04 01:35:09.846597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.846542 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x"] Nov 04 01:35:09.850047 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.850027 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-njn9x"] Nov 04 01:35:09.852216 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.852201 4852 scope.go:122] "RemoveContainer" containerID="4d3e9f4714237102b221f4ce40a934c4eebe7fa18ca6ef148f5fe30b0aed4a28" Nov 04 01:35:09.852464 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:35:09.852444 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d3e9f4714237102b221f4ce40a934c4eebe7fa18ca6ef148f5fe30b0aed4a28\": container with ID starting with 4d3e9f4714237102b221f4ce40a934c4eebe7fa18ca6ef148f5fe30b0aed4a28 not found: ID does not exist" containerID="4d3e9f4714237102b221f4ce40a934c4eebe7fa18ca6ef148f5fe30b0aed4a28" Nov 04 01:35:09.852526 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.852470 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d3e9f4714237102b221f4ce40a934c4eebe7fa18ca6ef148f5fe30b0aed4a28"} err="failed to get container status \"4d3e9f4714237102b221f4ce40a934c4eebe7fa18ca6ef148f5fe30b0aed4a28\": rpc error: code = NotFound desc = could not find container \"4d3e9f4714237102b221f4ce40a934c4eebe7fa18ca6ef148f5fe30b0aed4a28\": container with ID starting with 4d3e9f4714237102b221f4ce40a934c4eebe7fa18ca6ef148f5fe30b0aed4a28 not found: ID does not exist" Nov 04 01:35:09.861091 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.861071 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn"] Nov 04 01:35:09.865548 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.865530 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vgktn"] Nov 04 01:35:09.878739 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.878716 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv"] Nov 04 01:35:09.882712 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.882693 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-b48tv"] Nov 04 01:35:09.993826 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.993765 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" path="/var/lib/kubelet/pods/5f385f89-e8a1-42b1-a6e1-839aaf1c3530/volumes" Nov 04 01:35:09.994187 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.994175 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" path="/var/lib/kubelet/pods/7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d/volumes" Nov 04 01:35:09.994593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.994581 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" path="/var/lib/kubelet/pods/71f0b9b3-5fd3-4857-967f-a18a6fb481d7/volumes" Nov 04 01:35:09.995258 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.995244 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" path="/var/lib/kubelet/pods/c8ff057d-6968-4e4a-b0cf-3c5fbc325173/volumes" Nov 04 01:35:09.995643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:09.995631 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" path="/var/lib/kubelet/pods/f84bf0b4-1021-4c15-8ce0-e7b77b11f850/volumes" Nov 04 01:35:10.141619 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:10.135533 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:10.745114 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:10.745074 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:10.763741 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:10.763663 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4" podUID="c85127bb-d834-4223-9736-4416ec320c01" containerName="payload-processing" containerID="cri-o://bb81cf78b4d5345994bb207725ca4fb1b40355354cd4ab84528f1bba1ea90a4d" gracePeriod=25 Nov 04 01:35:10.804332 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:10.804280 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4" podUID="9b093e14-419d-48f2-9b07-978b6ad8ae72" containerName="payload-pre-processing" containerID="cri-o://fae9925231cbd60e4047d81d1548dd4dabc36c736e45c4c38439402e1ab70bfb" gracePeriod=25 Nov 04 01:35:11.081068 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.081044 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4" Nov 04 01:35:11.104558 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.104537 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4" Nov 04 01:35:11.149871 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.149101 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c85127bb-d834-4223-9736-4416ec320c01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85127bb-d834-4223-9736-4416ec320c01-plugins-config-volume\") pod \"c85127bb-d834-4223-9736-4416ec320c01\" (UID: \"c85127bb-d834-4223-9736-4416ec320c01\") " Nov 04 01:35:11.149871 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.149241 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c85127bb-d834-4223-9736-4416ec320c01-kube-api-access-5fvsg\" (UniqueName: \"kubernetes.io/projected/c85127bb-d834-4223-9736-4416ec320c01-kube-api-access-5fvsg\") pod \"c85127bb-d834-4223-9736-4416ec320c01\" (UID: \"c85127bb-d834-4223-9736-4416ec320c01\") " Nov 04 01:35:11.149871 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.149291 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b093e14-419d-48f2-9b07-978b6ad8ae72-kube-api-access-wvvr9\" (UniqueName: \"kubernetes.io/projected/9b093e14-419d-48f2-9b07-978b6ad8ae72-kube-api-access-wvvr9\") pod \"9b093e14-419d-48f2-9b07-978b6ad8ae72\" (UID: \"9b093e14-419d-48f2-9b07-978b6ad8ae72\") " Nov 04 01:35:11.149871 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.149342 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b093e14-419d-48f2-9b07-978b6ad8ae72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b093e14-419d-48f2-9b07-978b6ad8ae72-plugins-config-volume\") pod \"9b093e14-419d-48f2-9b07-978b6ad8ae72\" (UID: \"9b093e14-419d-48f2-9b07-978b6ad8ae72\") " Nov 04 01:35:11.150371 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.150343 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b093e14-419d-48f2-9b07-978b6ad8ae72-plugins-config-volume" pod "9b093e14-419d-48f2-9b07-978b6ad8ae72" (UID: "9b093e14-419d-48f2-9b07-978b6ad8ae72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:11.151903 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.150554 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b093e14-419d-48f2-9b07-978b6ad8ae72-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:11.151903 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.151050 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85127bb-d834-4223-9736-4416ec320c01-plugins-config-volume" pod "c85127bb-d834-4223-9736-4416ec320c01" (UID: "c85127bb-d834-4223-9736-4416ec320c01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:11.157991 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.157733 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b093e14-419d-48f2-9b07-978b6ad8ae72-kube-api-access-wvvr9" pod "9b093e14-419d-48f2-9b07-978b6ad8ae72" (UID: "9b093e14-419d-48f2-9b07-978b6ad8ae72"). InnerVolumeSpecName "kube-api-access-wvvr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:11.186521 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.169805 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85127bb-d834-4223-9736-4416ec320c01-kube-api-access-5fvsg" pod "c85127bb-d834-4223-9736-4416ec320c01" (UID: "c85127bb-d834-4223-9736-4416ec320c01"). InnerVolumeSpecName "kube-api-access-5fvsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:11.195240 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.195204 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:11.251138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.251101 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvvr9\" (UniqueName: \"kubernetes.io/projected/9b093e14-419d-48f2-9b07-978b6ad8ae72-kube-api-access-wvvr9\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:11.251138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.251128 4852 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85127bb-d834-4223-9736-4416ec320c01-plugins-config-volume\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:11.251138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.251138 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fvsg\" (UniqueName: \"kubernetes.io/projected/c85127bb-d834-4223-9736-4416ec320c01-kube-api-access-5fvsg\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:11.834378 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.834301 4852 generic.go:356] "Generic (PLEG): container finished" podID="c85127bb-d834-4223-9736-4416ec320c01" containerID="bb81cf78b4d5345994bb207725ca4fb1b40355354cd4ab84528f1bba1ea90a4d" exitCode=0 Nov 04 01:35:11.834378 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.834340 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4" Nov 04 01:35:11.834900 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.834395 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4" event={"ID":"c85127bb-d834-4223-9736-4416ec320c01","Type":"ContainerDied","Data":"bb81cf78b4d5345994bb207725ca4fb1b40355354cd4ab84528f1bba1ea90a4d"} Nov 04 01:35:11.834900 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.834441 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4" event={"ID":"c85127bb-d834-4223-9736-4416ec320c01","Type":"ContainerDied","Data":"2b8117a11661c60d91e6f43ea81a13537f729aefc03489dfa470f88edda02111"} Nov 04 01:35:11.834900 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.834478 4852 scope.go:122] "RemoveContainer" containerID="bb81cf78b4d5345994bb207725ca4fb1b40355354cd4ab84528f1bba1ea90a4d" Nov 04 01:35:11.835700 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.835676 4852 generic.go:356] "Generic (PLEG): container finished" podID="9b093e14-419d-48f2-9b07-978b6ad8ae72" containerID="fae9925231cbd60e4047d81d1548dd4dabc36c736e45c4c38439402e1ab70bfb" exitCode=0 Nov 04 01:35:11.835816 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.835706 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4" event={"ID":"9b093e14-419d-48f2-9b07-978b6ad8ae72","Type":"ContainerDied","Data":"fae9925231cbd60e4047d81d1548dd4dabc36c736e45c4c38439402e1ab70bfb"} Nov 04 01:35:11.835816 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.835723 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4" Nov 04 01:35:11.835816 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.835750 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4" event={"ID":"9b093e14-419d-48f2-9b07-978b6ad8ae72","Type":"ContainerDied","Data":"e2ea04fb31f3fdd65f7bf6e54aff45e578eafa5a89a781179c61cf9d0166bec0"} Nov 04 01:35:11.845432 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.845413 4852 scope.go:122] "RemoveContainer" containerID="bb81cf78b4d5345994bb207725ca4fb1b40355354cd4ab84528f1bba1ea90a4d" Nov 04 01:35:11.845705 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:35:11.845685 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb81cf78b4d5345994bb207725ca4fb1b40355354cd4ab84528f1bba1ea90a4d\": container with ID starting with bb81cf78b4d5345994bb207725ca4fb1b40355354cd4ab84528f1bba1ea90a4d not found: ID does not exist" containerID="bb81cf78b4d5345994bb207725ca4fb1b40355354cd4ab84528f1bba1ea90a4d" Nov 04 01:35:11.845800 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.845712 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb81cf78b4d5345994bb207725ca4fb1b40355354cd4ab84528f1bba1ea90a4d"} err="failed to get container status \"bb81cf78b4d5345994bb207725ca4fb1b40355354cd4ab84528f1bba1ea90a4d\": rpc error: code = NotFound desc = could not find container \"bb81cf78b4d5345994bb207725ca4fb1b40355354cd4ab84528f1bba1ea90a4d\": container with ID starting with bb81cf78b4d5345994bb207725ca4fb1b40355354cd4ab84528f1bba1ea90a4d not found: ID does not exist" Nov 04 01:35:11.845800 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.845731 4852 scope.go:122] "RemoveContainer" containerID="fae9925231cbd60e4047d81d1548dd4dabc36c736e45c4c38439402e1ab70bfb" Nov 04 01:35:11.854479 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.854456 4852 scope.go:122] "RemoveContainer" containerID="fae9925231cbd60e4047d81d1548dd4dabc36c736e45c4c38439402e1ab70bfb" Nov 04 01:35:11.854751 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:35:11.854729 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fae9925231cbd60e4047d81d1548dd4dabc36c736e45c4c38439402e1ab70bfb\": container with ID starting with fae9925231cbd60e4047d81d1548dd4dabc36c736e45c4c38439402e1ab70bfb not found: ID does not exist" containerID="fae9925231cbd60e4047d81d1548dd4dabc36c736e45c4c38439402e1ab70bfb" Nov 04 01:35:11.854834 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.854756 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fae9925231cbd60e4047d81d1548dd4dabc36c736e45c4c38439402e1ab70bfb"} err="failed to get container status \"fae9925231cbd60e4047d81d1548dd4dabc36c736e45c4c38439402e1ab70bfb\": rpc error: code = NotFound desc = could not find container \"fae9925231cbd60e4047d81d1548dd4dabc36c736e45c4c38439402e1ab70bfb\": container with ID starting with fae9925231cbd60e4047d81d1548dd4dabc36c736e45c4c38439402e1ab70bfb not found: ID does not exist" Nov 04 01:35:11.860937 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.860915 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4"] Nov 04 01:35:11.863846 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.863826 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016d3d-7469bfc947-lf7l4"] Nov 04 01:35:11.875208 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.875190 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4"] Nov 04 01:35:11.880873 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.880848 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-016d3d-767f4fcc46-cslc4"] Nov 04 01:35:11.993719 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.993696 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b093e14-419d-48f2-9b07-978b6ad8ae72" path="/var/lib/kubelet/pods/9b093e14-419d-48f2-9b07-978b6ad8ae72/volumes" Nov 04 01:35:11.994060 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:11.994044 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c85127bb-d834-4223-9736-4416ec320c01" path="/var/lib/kubelet/pods/c85127bb-d834-4223-9736-4416ec320c01/volumes" Nov 04 01:35:12.058750 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:12.058717 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:12.893392 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:12.893350 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:13.463387 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:13.463354 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845"] Nov 04 01:35:13.655404 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:13.655368 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:13.849583 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:13.848861 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:14.182692 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.182586 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:14.194639 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.194615 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z"] Nov 04 01:35:14.195022 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195006 4852 cpu_manager.go:414] "containerMap: removing container" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" Nov 04 01:35:14.195109 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195025 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" Nov 04 01:35:14.195109 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195044 4852 cpu_manager.go:414] "containerMap: removing container" podUID="719d829c-c70a-4bdc-a3d2-5b4cb52a45da" containerName="istio-proxy" Nov 04 01:35:14.195109 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195055 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="719d829c-c70a-4bdc-a3d2-5b4cb52a45da" containerName="istio-proxy" Nov 04 01:35:14.195109 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195065 4852 cpu_manager.go:414] "containerMap: removing container" podUID="614a961a-dc3c-41ae-963f-53dfe934eda0" containerName="payload-pre-processing" Nov 04 01:35:14.195109 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195072 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="614a961a-dc3c-41ae-963f-53dfe934eda0" containerName="payload-pre-processing" Nov 04 01:35:14.195109 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195086 4852 cpu_manager.go:414] "containerMap: removing container" podUID="39133c56-c8e4-4b42-9df6-7dc091b4f09d" containerName="payload-processing" Nov 04 01:35:14.195109 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195093 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39133c56-c8e4-4b42-9df6-7dc091b4f09d" containerName="payload-processing" Nov 04 01:35:14.195109 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195103 4852 cpu_manager.go:414] "containerMap: removing container" podUID="2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4" containerName="payload-processing" Nov 04 01:35:14.195109 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195109 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4" containerName="payload-processing" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195123 4852 cpu_manager.go:414] "containerMap: removing container" podUID="57537b5d-473c-41da-9e67-6c848a66fee7" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195130 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57537b5d-473c-41da-9e67-6c848a66fee7" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195144 4852 cpu_manager.go:414] "containerMap: removing container" podUID="0e92e697-d5a8-4f9c-9716-697d3f9e2546" containerName="payload-pre-processing" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195152 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e92e697-d5a8-4f9c-9716-697d3f9e2546" containerName="payload-pre-processing" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195165 4852 cpu_manager.go:414] "containerMap: removing container" podUID="1efe83d4-74d6-431e-91fd-314ce0f5d116" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195172 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1efe83d4-74d6-431e-91fd-314ce0f5d116" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195183 4852 cpu_manager.go:414] "containerMap: removing container" podUID="1efe83d4-74d6-431e-91fd-314ce0f5d116" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195190 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1efe83d4-74d6-431e-91fd-314ce0f5d116" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195206 4852 cpu_manager.go:414] "containerMap: removing container" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195213 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195225 4852 cpu_manager.go:414] "containerMap: removing container" podUID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195232 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195243 4852 cpu_manager.go:414] "containerMap: removing container" podUID="e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed" containerName="payload-pre-processing" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195250 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed" containerName="payload-pre-processing" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195259 4852 cpu_manager.go:414] "containerMap: removing container" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195266 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195276 4852 cpu_manager.go:414] "containerMap: removing container" podUID="7af43160-27ea-4e98-8e54-46e9a0adc36c" containerName="payload-pre-processing" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195283 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af43160-27ea-4e98-8e54-46e9a0adc36c" containerName="payload-pre-processing" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195296 4852 cpu_manager.go:414] "containerMap: removing container" podUID="456612ef-2ef3-4b16-bab3-1277c84311c5" containerName="payload-processing" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195302 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="456612ef-2ef3-4b16-bab3-1277c84311c5" containerName="payload-processing" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195317 4852 cpu_manager.go:414] "containerMap: removing container" podUID="d31799de-2b0d-4f2e-839e-058668f304c6" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195324 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d31799de-2b0d-4f2e-839e-058668f304c6" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195333 4852 cpu_manager.go:414] "containerMap: removing container" podUID="ed9eda72-0b3b-4933-bb4d-83d8eee5c20e" containerName="payload-processing" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195341 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed9eda72-0b3b-4933-bb4d-83d8eee5c20e" containerName="payload-processing" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195352 4852 cpu_manager.go:414] "containerMap: removing container" podUID="f3ab7ef8-0104-4e1f-8327-41964669c3b3" containerName="payload-pre-processing" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195358 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3ab7ef8-0104-4e1f-8327-41964669c3b3" containerName="payload-pre-processing" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195368 4852 cpu_manager.go:414] "containerMap: removing container" podUID="6604612f-7353-40ab-8a1b-29b19b3395f0" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195374 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6604612f-7353-40ab-8a1b-29b19b3395f0" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195383 4852 cpu_manager.go:414] "containerMap: removing container" podUID="6604612f-7353-40ab-8a1b-29b19b3395f0" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195391 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6604612f-7353-40ab-8a1b-29b19b3395f0" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195404 4852 cpu_manager.go:414] "containerMap: removing container" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195411 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195425 4852 cpu_manager.go:414] "containerMap: removing container" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195431 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195441 4852 cpu_manager.go:414] "containerMap: removing container" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195447 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195456 4852 cpu_manager.go:414] "containerMap: removing container" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195463 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195475 4852 cpu_manager.go:414] "containerMap: removing container" podUID="9febb536-9d73-41c8-8a6e-5b2aa1446825" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195482 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9febb536-9d73-41c8-8a6e-5b2aa1446825" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195495 4852 cpu_manager.go:414] "containerMap: removing container" podUID="852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195502 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195511 4852 cpu_manager.go:414] "containerMap: removing container" podUID="a5c642db-a1e8-4839-9fa7-1821588ecb15" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195518 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c642db-a1e8-4839-9fa7-1821588ecb15" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195529 4852 cpu_manager.go:414] "containerMap: removing container" podUID="d26cecde-b16d-4fe1-add0-41a6d0a5301a" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195536 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26cecde-b16d-4fe1-add0-41a6d0a5301a" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195546 4852 cpu_manager.go:414] "containerMap: removing container" podUID="c85127bb-d834-4223-9736-4416ec320c01" containerName="payload-processing" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195554 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c85127bb-d834-4223-9736-4416ec320c01" containerName="payload-processing" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195586 4852 cpu_manager.go:414] "containerMap: removing container" podUID="9b093e14-419d-48f2-9b07-978b6ad8ae72" containerName="payload-pre-processing" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195593 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b093e14-419d-48f2-9b07-978b6ad8ae72" containerName="payload-pre-processing" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195609 4852 cpu_manager.go:414] "containerMap: removing container" podUID="ab9122ac-8874-4a64-9137-a7fe056db11e" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195616 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab9122ac-8874-4a64-9137-a7fe056db11e" containerName="istio-proxy" Nov 04 01:35:14.195605 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195628 4852 cpu_manager.go:414] "containerMap: removing container" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195637 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195647 4852 cpu_manager.go:414] "containerMap: removing container" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195654 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195664 4852 cpu_manager.go:414] "containerMap: removing container" podUID="7619e54f-752b-428f-b955-b2dec4471c5b" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195671 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7619e54f-752b-428f-b955-b2dec4471c5b" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195680 4852 cpu_manager.go:414] "containerMap: removing container" podUID="7d5e42b9-cad8-4145-8824-f972cc7d6505" containerName="maas-api" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195687 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d5e42b9-cad8-4145-8824-f972cc7d6505" containerName="maas-api" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195697 4852 cpu_manager.go:414] "containerMap: removing container" podUID="9961d5eb-fd3c-4651-96a8-861577e1ad21" containerName="maas-api" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195704 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9961d5eb-fd3c-4651-96a8-861577e1ad21" containerName="maas-api" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195721 4852 cpu_manager.go:414] "containerMap: removing container" podUID="68127737-8431-4191-ad9a-d16e080a2ec5" containerName="payload-pre-processing" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195729 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68127737-8431-4191-ad9a-d16e080a2ec5" containerName="payload-pre-processing" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195742 4852 cpu_manager.go:414] "containerMap: removing container" podUID="d31799de-2b0d-4f2e-839e-058668f304c6" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195748 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d31799de-2b0d-4f2e-839e-058668f304c6" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195759 4852 cpu_manager.go:414] "containerMap: removing container" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195767 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195780 4852 cpu_manager.go:414] "containerMap: removing container" podUID="31ae07d8-c2d3-4732-83ff-b9583ec0fba8" containerName="maas-api" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195786 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ae07d8-c2d3-4732-83ff-b9583ec0fba8" containerName="maas-api" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195799 4852 cpu_manager.go:414] "containerMap: removing container" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195805 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195816 4852 cpu_manager.go:414] "containerMap: removing container" podUID="baefa3c0-28be-428a-9591-f0ca1f07fafa" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195823 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baefa3c0-28be-428a-9591-f0ca1f07fafa" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195832 4852 cpu_manager.go:414] "containerMap: removing container" podUID="9febb536-9d73-41c8-8a6e-5b2aa1446825" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195839 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9febb536-9d73-41c8-8a6e-5b2aa1446825" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195962 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ae07d8-c2d3-4732-83ff-b9583ec0fba8" containerName="maas-api" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195977 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="c69e53e0-661a-4a14-94de-d4450a554c80" containerName="main" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195988 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="baefa3c0-28be-428a-9591-f0ca1f07fafa" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.195996 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="9febb536-9d73-41c8-8a6e-5b2aa1446825" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196006 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196015 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="719d829c-c70a-4bdc-a3d2-5b4cb52a45da" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196024 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="614a961a-dc3c-41ae-963f-53dfe934eda0" containerName="payload-pre-processing" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196032 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="39133c56-c8e4-4b42-9df6-7dc091b4f09d" containerName="payload-processing" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196043 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d752c6b-ef0d-4eeb-a1de-10f3e889f0f4" containerName="payload-processing" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196053 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e92e697-d5a8-4f9c-9716-697d3f9e2546" containerName="payload-pre-processing" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196065 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="1efe83d4-74d6-431e-91fd-314ce0f5d116" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196073 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="1efe83d4-74d6-431e-91fd-314ce0f5d116" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196084 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="7025b15d-c99e-4ccb-98a2-2a6f4cc69a1d" containerName="main" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196094 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="cae606b8-4a95-4046-a9eb-3df149dcf7a8" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196102 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="e15e5a9d-84ef-4e2b-9b9d-4b5eb1b767ed" containerName="payload-pre-processing" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196112 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8ff057d-6968-4e4a-b0cf-3c5fbc325173" containerName="main" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196122 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af43160-27ea-4e98-8e54-46e9a0adc36c" containerName="payload-pre-processing" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196129 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="456612ef-2ef3-4b16-bab3-1277c84311c5" containerName="payload-processing" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196137 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="d31799de-2b0d-4f2e-839e-058668f304c6" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196145 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed9eda72-0b3b-4933-bb4d-83d8eee5c20e" containerName="payload-processing" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196152 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3ab7ef8-0104-4e1f-8327-41964669c3b3" containerName="payload-pre-processing" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196160 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="6604612f-7353-40ab-8a1b-29b19b3395f0" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196167 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="6604612f-7353-40ab-8a1b-29b19b3395f0" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196178 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="9211f266-8d05-4bd8-9947-7893a9c0634b" containerName="main" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196187 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce1f2a5-3954-4044-a85e-2761a6b2088e" containerName="main" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196195 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="761be4d8-0736-4a22-bd18-2729dbc85a11" containerName="main" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196203 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196213 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="852ff9e8-47a9-4144-b0a1-ff5f9c9cb6e0" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196220 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c642db-a1e8-4839-9fa7-1821588ecb15" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196228 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26cecde-b16d-4fe1-add0-41a6d0a5301a" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196236 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="c85127bb-d834-4223-9736-4416ec320c01" containerName="payload-processing" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196247 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b093e14-419d-48f2-9b07-978b6ad8ae72" containerName="payload-pre-processing" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196255 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab9122ac-8874-4a64-9137-a7fe056db11e" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196263 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f385f89-e8a1-42b1-a6e1-839aaf1c3530" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196271 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="f84bf0b4-1021-4c15-8ce0-e7b77b11f850" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196279 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="7619e54f-752b-428f-b955-b2dec4471c5b" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196287 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d5e42b9-cad8-4145-8824-f972cc7d6505" containerName="maas-api" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196296 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="9961d5eb-fd3c-4651-96a8-861577e1ad21" containerName="maas-api" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196305 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="68127737-8431-4191-ad9a-d16e080a2ec5" containerName="payload-pre-processing" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196314 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="d31799de-2b0d-4f2e-839e-058668f304c6" containerName="istio-proxy" Nov 04 01:35:14.197938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.196322 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="71f0b9b3-5fd3-4857-967f-a18a6fb481d7" containerName="istio-proxy" Nov 04 01:35:14.200179 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.199284 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.218824 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.218800 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z"] Nov 04 01:35:14.245693 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.245666 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl"] Nov 04 01:35:14.246040 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.246024 4852 cpu_manager.go:414] "containerMap: removing container" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" Nov 04 01:35:14.246040 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.246040 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" Nov 04 01:35:14.246172 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.246121 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="9febb536-9d73-41c8-8a6e-5b2aa1446825" containerName="istio-proxy" Nov 04 01:35:14.246172 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.246130 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="c72aca29-e3f2-4a8f-abea-8841337cc569" containerName="istio-proxy" Nov 04 01:35:14.247927 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.247906 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.269646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.269622 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl"] Nov 04 01:35:14.293971 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.293926 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbp2n\" (UniqueName: \"kubernetes.io/projected/4bdbb81d-304f-4cb6-8282-813bf09b8699-kube-api-access-gbp2n\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.294120 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.293987 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.294120 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.294092 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.294231 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.294145 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.294231 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.294174 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.294231 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.294205 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.294394 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.294234 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.294394 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.294268 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.294394 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.294297 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.294394 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.294366 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.294655 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.294399 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.294655 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.294424 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.294655 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.294449 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.294655 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.294488 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdbb81d-304f-4cb6-8282-813bf09b8699-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.294655 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.294528 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.294655 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.294587 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.294655 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.294622 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.294655 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.294651 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4kdz\" (UniqueName: \"kubernetes.io/projected/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-kube-api-access-n4kdz\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.395834 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.395808 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbp2n\" (UniqueName: \"kubernetes.io/projected/4bdbb81d-304f-4cb6-8282-813bf09b8699-kube-api-access-gbp2n\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.396008 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.395847 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.396008 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.395869 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.396008 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.395897 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.396008 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.395925 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.396008 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.395968 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.396008 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.395994 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.396328 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.396023 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.396328 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.396049 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.396328 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.396109 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.396328 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.396140 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.396328 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.396165 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.396328 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.396192 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.396328 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.396221 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdbb81d-304f-4cb6-8282-813bf09b8699-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.396328 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.396258 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.396328 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.396291 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.396328 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.396323 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.397089 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.396349 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4kdz\" (UniqueName: \"kubernetes.io/projected/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-kube-api-access-n4kdz\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.397089 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.396379 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.397089 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.396395 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.397089 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.396657 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.397089 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.396805 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.397089 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.396928 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.397413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.397197 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.397413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.397224 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.397600 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.397550 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.397915 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.397891 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.398077 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.398041 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdbb81d-304f-4cb6-8282-813bf09b8699-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.399201 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.399173 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.399865 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.399844 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.400084 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.400053 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.400193 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.400175 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.405133 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.405107 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.405500 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.405478 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbp2n\" (UniqueName: \"kubernetes.io/projected/4bdbb81d-304f-4cb6-8282-813bf09b8699-kube-api-access-gbp2n\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.406429 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.406409 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.406533 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.406516 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4kdz\" (UniqueName: \"kubernetes.io/projected/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-kube-api-access-n4kdz\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.510604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.510555 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:14.568775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.568485 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:14.734640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.734591 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:14.790515 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.790493 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z"] Nov 04 01:35:14.793059 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:35:14.793036 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c2e335_58f4_4c3f_bea2_14a6dc9865ff.slice/crio-dd7e3c1b0c9cae8eaad35de506dc47b18e89552ffebf39c02e2ac64e04b21f91 WatchSource:0}: Error finding container dd7e3c1b0c9cae8eaad35de506dc47b18e89552ffebf39c02e2ac64e04b21f91: Status 404 returned error can't find the container with id dd7e3c1b0c9cae8eaad35de506dc47b18e89552ffebf39c02e2ac64e04b21f91 Nov 04 01:35:14.795276 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.795244 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:14.795357 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.795319 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:14.795357 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.795348 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:14.869503 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.869407 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl"] Nov 04 01:35:14.873248 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.872998 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" event={"ID":"26c2e335-58f4-4c3f-bea2-14a6dc9865ff","Type":"ContainerStarted","Data":"dd7e3c1b0c9cae8eaad35de506dc47b18e89552ffebf39c02e2ac64e04b21f91"} Nov 04 01:35:14.878165 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:35:14.878116 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bdbb81d_304f_4cb6_8282_813bf09b8699.slice/crio-e6665f8c8440b1f2078d728b6b836d9aeede193bb9e733550ff7e55610117fe2 WatchSource:0}: Error finding container e6665f8c8440b1f2078d728b6b836d9aeede193bb9e733550ff7e55610117fe2: Status 404 returned error can't find the container with id e6665f8c8440b1f2078d728b6b836d9aeede193bb9e733550ff7e55610117fe2 Nov 04 01:35:14.881598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.881211 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:14.881598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.881286 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:14.881598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.881323 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:14.900154 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:14.900081 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:15.073716 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:15.073627 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:15.308408 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:15.308374 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:15.452364 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:15.452283 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:15.749116 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:15.748398 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj"] Nov 04 01:35:15.749116 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:15.748673 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" podUID="b0515d69-ac7d-4d43-a316-58ccc63b757a" containerName="istio-proxy" containerID="cri-o://784c828947250e74ae3aecf94b2e833fcd4ed99ecebc4a02f1cb562ecc8e840c" gracePeriod=30 Nov 04 01:35:15.911607 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:15.905771 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" event={"ID":"26c2e335-58f4-4c3f-bea2-14a6dc9865ff","Type":"ContainerStarted","Data":"7d97b84216b7afaf499b293ba6fa8991b44e088053721eee46814e8ceec354fa"} Nov 04 01:35:15.923216 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:15.921316 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:15.923216 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:15.922309 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" event={"ID":"4bdbb81d-304f-4cb6-8282-813bf09b8699","Type":"ContainerStarted","Data":"53d99116f09a0c9f7dd551199583cc2094cb8b161cb4b19333c757d6712ab626"} Nov 04 01:35:15.923216 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:15.922361 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" event={"ID":"4bdbb81d-304f-4cb6-8282-813bf09b8699","Type":"ContainerStarted","Data":"e6665f8c8440b1f2078d728b6b836d9aeede193bb9e733550ff7e55610117fe2"} Nov 04 01:35:15.947065 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:15.947011 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" podStartSLOduration=1.946996472 podStartE2EDuration="1.946996472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:15.944947675 +0000 UTC m=+2198.060057795" watchObservedRunningTime="2026-11-04 01:35:15.946996472 +0000 UTC m=+2198.062106589" Nov 04 01:35:16.094401 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:16.094353 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:16.496455 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:16.496336 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:16.511082 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:16.511048 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:16.512695 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:16.512663 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:35:16.512899 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:16.512728 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" podUID="26c2e335-58f4-4c3f-bea2-14a6dc9865ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 04 01:35:16.576589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:16.574954 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:16.585176 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:16.585140 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:35:16.585447 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:16.585417 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" podUID="4bdbb81d-304f-4cb6-8282-813bf09b8699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 04 01:35:16.807973 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:16.807934 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:17.007238 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:17.007198 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:17.511646 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:17.511613 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:35:17.512159 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:17.512133 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" podUID="26c2e335-58f4-4c3f-bea2-14a6dc9865ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 04 01:35:17.575591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:17.571976 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:35:17.575591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:17.572040 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" podUID="4bdbb81d-304f-4cb6-8282-813bf09b8699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 04 01:35:17.702008 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:17.695747 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:17.718865 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:17.714791 4852 patch_prober.go:29] interesting pod/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:35:17.718865 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:17.714940 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" podUID="b0515d69-ac7d-4d43-a316-58ccc63b757a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:35:18.511116 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:18.511069 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:35:18.511313 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:18.511132 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" podUID="26c2e335-58f4-4c3f-bea2-14a6dc9865ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 04 01:35:18.571591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:18.568987 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:35:18.571591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:18.569052 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" podUID="4bdbb81d-304f-4cb6-8282-813bf09b8699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 04 01:35:18.697588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:18.691362 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:19.397626 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:19.397546 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" podStartSLOduration=5.397503257 podStartE2EDuration="5.397503257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:15.976932308 +0000 UTC m=+2198.092042431" watchObservedRunningTime="2026-11-04 01:35:19.397503257 +0000 UTC m=+2201.512613374" Nov 04 01:35:19.400740 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:19.400712 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw"] Nov 04 01:35:19.413879 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:19.412620 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw"] Nov 04 01:35:19.413879 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:19.412732 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw" Nov 04 01:35:19.437585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:19.437552 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:35:19.437709 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:19.437644 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:35:19.503906 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:19.502179 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46120899-8e38-4afd-8e30-787092a30fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-616cb1ce-k7mbw\" (UID: \"46120899-8e38-4afd-8e30-787092a30fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw" Nov 04 01:35:19.503906 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:19.502291 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46120899-8e38-4afd-8e30-787092a30fc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-616cb1ce-k7mbw\" (UID: \"46120899-8e38-4afd-8e30-787092a30fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw" Nov 04 01:35:19.503906 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:19.502345 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flrrh\" (UniqueName: \"kubernetes.io/projected/46120899-8e38-4afd-8e30-787092a30fc8-kube-api-access-flrrh\") pod \"maas-api-revoke-sub-616cb1ce-k7mbw\" (UID: \"46120899-8e38-4afd-8e30-787092a30fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw" Nov 04 01:35:19.511287 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:19.511169 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:35:19.511287 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:19.511259 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" podUID="26c2e335-58f4-4c3f-bea2-14a6dc9865ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 04 01:35:19.580602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:19.577089 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:35:19.580602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:19.577145 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" podUID="4bdbb81d-304f-4cb6-8282-813bf09b8699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 04 01:35:19.604260 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:19.604210 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flrrh\" (UniqueName: \"kubernetes.io/projected/46120899-8e38-4afd-8e30-787092a30fc8-kube-api-access-flrrh\") pod \"maas-api-revoke-sub-616cb1ce-k7mbw\" (UID: \"46120899-8e38-4afd-8e30-787092a30fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw" Nov 04 01:35:19.605013 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:19.604990 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46120899-8e38-4afd-8e30-787092a30fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-616cb1ce-k7mbw\" (UID: \"46120899-8e38-4afd-8e30-787092a30fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw" Nov 04 01:35:19.605876 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:19.605834 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46120899-8e38-4afd-8e30-787092a30fc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-616cb1ce-k7mbw\" (UID: \"46120899-8e38-4afd-8e30-787092a30fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw" Nov 04 01:35:19.606359 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:19.606339 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46120899-8e38-4afd-8e30-787092a30fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-616cb1ce-k7mbw\" (UID: \"46120899-8e38-4afd-8e30-787092a30fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw" Nov 04 01:35:19.609972 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:19.609946 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46120899-8e38-4afd-8e30-787092a30fc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-616cb1ce-k7mbw\" (UID: \"46120899-8e38-4afd-8e30-787092a30fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw" Nov 04 01:35:19.616807 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:19.616791 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flrrh\" (UniqueName: \"kubernetes.io/projected/46120899-8e38-4afd-8e30-787092a30fc8-kube-api-access-flrrh\") pod \"maas-api-revoke-sub-616cb1ce-k7mbw\" (UID: \"46120899-8e38-4afd-8e30-787092a30fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw" Nov 04 01:35:19.718759 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:19.718656 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:19.731827 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:19.731784 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw" Nov 04 01:35:20.108593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:20.095849 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw"] Nov 04 01:35:20.110856 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:35:20.110825 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46120899_8e38_4afd_8e30_787092a30fc8.slice/crio-17f5fa33bf5aaf0e1ef4ffaf382a4cbcb3d55511009f3fa42f251b65ef6f9b13 WatchSource:0}: Error finding container 17f5fa33bf5aaf0e1ef4ffaf382a4cbcb3d55511009f3fa42f251b65ef6f9b13: Status 404 returned error can't find the container with id 17f5fa33bf5aaf0e1ef4ffaf382a4cbcb3d55511009f3fa42f251b65ef6f9b13 Nov 04 01:35:20.511223 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:20.511196 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:35:20.511445 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:20.511411 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" podUID="26c2e335-58f4-4c3f-bea2-14a6dc9865ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 04 01:35:20.576541 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:20.576509 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:35:20.576800 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:20.576775 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" podUID="4bdbb81d-304f-4cb6-8282-813bf09b8699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 04 01:35:20.640615 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:20.639770 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:20.953910 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:20.953860 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw" event={"ID":"46120899-8e38-4afd-8e30-787092a30fc8","Type":"ContainerStarted","Data":"a995b495234bda418206abb2d98a21c85c94ea2d6a9e1503fd1b14d0cf618659"} Nov 04 01:35:20.954105 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:20.953938 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw" event={"ID":"46120899-8e38-4afd-8e30-787092a30fc8","Type":"ContainerStarted","Data":"17f5fa33bf5aaf0e1ef4ffaf382a4cbcb3d55511009f3fa42f251b65ef6f9b13"} Nov 04 01:35:20.955977 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:20.955950 4852 generic.go:356] "Generic (PLEG): container finished" podID="b0515d69-ac7d-4d43-a316-58ccc63b757a" containerID="784c828947250e74ae3aecf94b2e833fcd4ed99ecebc4a02f1cb562ecc8e840c" exitCode=0 Nov 04 01:35:20.956093 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:20.956020 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" event={"ID":"b0515d69-ac7d-4d43-a316-58ccc63b757a","Type":"ContainerDied","Data":"784c828947250e74ae3aecf94b2e833fcd4ed99ecebc4a02f1cb562ecc8e840c"} Nov 04 01:35:20.956093 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:20.956076 4852 scope.go:122] "RemoveContainer" containerID="040ea5bf349d71a7e21d814bbf7efe6e5e85b338a6831db42692d76f4766b5ac" Nov 04 01:35:21.016766 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.013348 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw" podStartSLOduration=2.013327476 podStartE2EDuration="2.013327476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:21.012922113 +0000 UTC m=+2203.128032232" watchObservedRunningTime="2026-11-04 01:35:21.013327476 +0000 UTC m=+2203.128437595" Nov 04 01:35:21.212848 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.212822 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:35:21.332977 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.332883 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-podinfo\") pod \"b0515d69-ac7d-4d43-a316-58ccc63b757a\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " Nov 04 01:35:21.332977 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.332939 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0515d69-ac7d-4d43-a316-58ccc63b757a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0515d69-ac7d-4d43-a316-58ccc63b757a-istiod-ca-cert\") pod \"b0515d69-ac7d-4d43-a316-58ccc63b757a\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " Nov 04 01:35:21.332977 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.332966 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-workload-socket\") pod \"b0515d69-ac7d-4d43-a316-58ccc63b757a\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " Nov 04 01:35:21.333208 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.333098 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-credential-socket\") pod \"b0515d69-ac7d-4d43-a316-58ccc63b757a\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " Nov 04 01:35:21.333208 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.333165 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-data\") pod \"b0515d69-ac7d-4d43-a316-58ccc63b757a\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " Nov 04 01:35:21.333295 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.333211 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-workload-certs\") pod \"b0515d69-ac7d-4d43-a316-58ccc63b757a\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " Nov 04 01:35:21.333295 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.333255 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-workload-socket" pod "b0515d69-ac7d-4d43-a316-58ccc63b757a" (UID: "b0515d69-ac7d-4d43-a316-58ccc63b757a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:21.333295 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.333272 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0515d69-ac7d-4d43-a316-58ccc63b757a-kube-api-access-zlnpf\" (UniqueName: \"kubernetes.io/projected/b0515d69-ac7d-4d43-a316-58ccc63b757a-kube-api-access-zlnpf\") pod \"b0515d69-ac7d-4d43-a316-58ccc63b757a\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " Nov 04 01:35:21.333447 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.333326 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-token\" (UniqueName: \"kubernetes.io/projected/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-token\") pod \"b0515d69-ac7d-4d43-a316-58ccc63b757a\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " Nov 04 01:35:21.333447 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.333398 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-envoy\") pod \"b0515d69-ac7d-4d43-a316-58ccc63b757a\" (UID: \"b0515d69-ac7d-4d43-a316-58ccc63b757a\") " Nov 04 01:35:21.333558 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.333499 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-workload-certs" pod "b0515d69-ac7d-4d43-a316-58ccc63b757a" (UID: "b0515d69-ac7d-4d43-a316-58ccc63b757a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:21.333558 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.333530 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-credential-socket" pod "b0515d69-ac7d-4d43-a316-58ccc63b757a" (UID: "b0515d69-ac7d-4d43-a316-58ccc63b757a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:21.333558 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.333549 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-data" pod "b0515d69-ac7d-4d43-a316-58ccc63b757a" (UID: "b0515d69-ac7d-4d43-a316-58ccc63b757a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:21.333814 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.333640 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0515d69-ac7d-4d43-a316-58ccc63b757a-istiod-ca-cert" pod "b0515d69-ac7d-4d43-a316-58ccc63b757a" (UID: "b0515d69-ac7d-4d43-a316-58ccc63b757a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:21.333959 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.333935 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:21.334052 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.333962 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:21.334052 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.333976 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0515d69-ac7d-4d43-a316-58ccc63b757a-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:21.334052 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.333991 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:21.334052 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.334005 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:21.336053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.336030 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0515d69-ac7d-4d43-a316-58ccc63b757a-kube-api-access-zlnpf" pod "b0515d69-ac7d-4d43-a316-58ccc63b757a" (UID: "b0515d69-ac7d-4d43-a316-58ccc63b757a"). InnerVolumeSpecName "kube-api-access-zlnpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:21.336189 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.336134 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-envoy" pod "b0515d69-ac7d-4d43-a316-58ccc63b757a" (UID: "b0515d69-ac7d-4d43-a316-58ccc63b757a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:21.336240 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.336223 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-podinfo" pod "b0515d69-ac7d-4d43-a316-58ccc63b757a" (UID: "b0515d69-ac7d-4d43-a316-58ccc63b757a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:21.336418 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.336403 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-token" pod "b0515d69-ac7d-4d43-a316-58ccc63b757a" (UID: "b0515d69-ac7d-4d43-a316-58ccc63b757a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:21.440093 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.437453 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:21.440093 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.437498 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlnpf\" (UniqueName: \"kubernetes.io/projected/b0515d69-ac7d-4d43-a316-58ccc63b757a-kube-api-access-zlnpf\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:21.440093 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.437514 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:21.440093 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.437528 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0515d69-ac7d-4d43-a316-58ccc63b757a-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:21.489925 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.489890 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:21.508626 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.508589 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p"] Nov 04 01:35:21.511541 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.511504 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:35:21.511714 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.511554 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" podUID="26c2e335-58f4-4c3f-bea2-14a6dc9865ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 04 01:35:21.517994 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.517970 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6"] Nov 04 01:35:21.534204 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.534177 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb"] Nov 04 01:35:21.570068 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.569604 4852 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:35:21.570068 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.569684 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" podUID="4bdbb81d-304f-4cb6-8282-813bf09b8699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 04 01:35:21.767140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.767110 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:21.965311 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.965273 4852 generic.go:356] "Generic (PLEG): container finished" podID="46120899-8e38-4afd-8e30-787092a30fc8" containerID="a995b495234bda418206abb2d98a21c85c94ea2d6a9e1503fd1b14d0cf618659" exitCode=0 Nov 04 01:35:21.965495 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.965370 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw" event={"ID":"46120899-8e38-4afd-8e30-787092a30fc8","Type":"ContainerDied","Data":"a995b495234bda418206abb2d98a21c85c94ea2d6a9e1503fd1b14d0cf618659"} Nov 04 01:35:21.967329 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.967308 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" Nov 04 01:35:21.967329 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.967313 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj" event={"ID":"b0515d69-ac7d-4d43-a316-58ccc63b757a","Type":"ContainerDied","Data":"3ed4d6d302faba91f9568d1be18a546f25eb932e0ce4e0decae3dd03dd739f50"} Nov 04 01:35:21.967627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:21.967375 4852 scope.go:122] "RemoveContainer" containerID="784c828947250e74ae3aecf94b2e833fcd4ed99ecebc4a02f1cb562ecc8e840c" Nov 04 01:35:22.011951 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:22.011842 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj"] Nov 04 01:35:22.016069 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:22.016046 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-016d3d-openshift-default-54957cc7c-4qkbj"] Nov 04 01:35:22.102618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:22.102504 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:22.268362 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:22.268317 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:22.541625 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:22.541589 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:22.542944 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:22.542917 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:22.544243 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:22.543909 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:22.583803 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:22.583767 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:22.683065 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:22.683026 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:22.685048 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:22.684038 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:22.685478 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:22.685454 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:22.787711 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:22.787682 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:23.127702 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:23.127679 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw" Nov 04 01:35:23.266058 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:23.266018 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46120899-8e38-4afd-8e30-787092a30fc8-kube-api-access-flrrh\" (UniqueName: \"kubernetes.io/projected/46120899-8e38-4afd-8e30-787092a30fc8-kube-api-access-flrrh\") pod \"46120899-8e38-4afd-8e30-787092a30fc8\" (UID: \"46120899-8e38-4afd-8e30-787092a30fc8\") " Nov 04 01:35:23.266240 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:23.266078 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46120899-8e38-4afd-8e30-787092a30fc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46120899-8e38-4afd-8e30-787092a30fc8-maas-api-xks-ca\") pod \"46120899-8e38-4afd-8e30-787092a30fc8\" (UID: \"46120899-8e38-4afd-8e30-787092a30fc8\") " Nov 04 01:35:23.266240 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:23.266166 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46120899-8e38-4afd-8e30-787092a30fc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46120899-8e38-4afd-8e30-787092a30fc8-maas-api-openshift-ca\") pod \"46120899-8e38-4afd-8e30-787092a30fc8\" (UID: \"46120899-8e38-4afd-8e30-787092a30fc8\") " Nov 04 01:35:23.266507 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:23.266483 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46120899-8e38-4afd-8e30-787092a30fc8-maas-api-openshift-ca" pod "46120899-8e38-4afd-8e30-787092a30fc8" (UID: "46120899-8e38-4afd-8e30-787092a30fc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:23.268242 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:23.268222 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46120899-8e38-4afd-8e30-787092a30fc8-maas-api-xks-ca" pod "46120899-8e38-4afd-8e30-787092a30fc8" (UID: "46120899-8e38-4afd-8e30-787092a30fc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:23.268321 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:23.268259 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46120899-8e38-4afd-8e30-787092a30fc8-kube-api-access-flrrh" pod "46120899-8e38-4afd-8e30-787092a30fc8" (UID: "46120899-8e38-4afd-8e30-787092a30fc8"). InnerVolumeSpecName "kube-api-access-flrrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:23.368491 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:23.368468 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flrrh\" (UniqueName: \"kubernetes.io/projected/46120899-8e38-4afd-8e30-787092a30fc8-kube-api-access-flrrh\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:23.368491 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:23.368492 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46120899-8e38-4afd-8e30-787092a30fc8-maas-api-xks-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:23.368654 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:23.368502 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46120899-8e38-4afd-8e30-787092a30fc8-maas-api-openshift-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:23.474895 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:23.474867 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:23.639076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:23.638968 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:23.991634 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:23.991518 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw" Nov 04 01:35:23.998469 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:23.998437 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0515d69-ac7d-4d43-a316-58ccc63b757a" path="/var/lib/kubelet/pods/b0515d69-ac7d-4d43-a316-58ccc63b757a/volumes" Nov 04 01:35:23.998928 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:23.998910 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw" event={"ID":"46120899-8e38-4afd-8e30-787092a30fc8","Type":"ContainerDied","Data":"17f5fa33bf5aaf0e1ef4ffaf382a4cbcb3d55511009f3fa42f251b65ef6f9b13"} Nov 04 01:35:23.998975 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:23.998956 4852 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17f5fa33bf5aaf0e1ef4ffaf382a4cbcb3d55511009f3fa42f251b65ef6f9b13" Nov 04 01:35:24.273056 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:24.273032 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:24.412722 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:24.412692 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:25.176588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:25.176224 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:25.913089 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:25.913034 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:26.658878 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:26.658848 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:35:26.743588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:26.730351 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:27.415708 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:27.415651 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:27.791207 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:27.791166 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:27.955018 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:27.954972 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:28.485490 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:28.485441 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" containerID="cri-o://e3260569ca48be59369e194750aa3a7ffc4a75f1c966ad60618289acd6c9e8f1" gracePeriod=45 Nov 04 01:35:28.840172 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:28.840142 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:35:28.947946 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:28.947046 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-tmp-dir\") pod \"790de7b2-be85-47bb-b2f2-d21b1422bac9\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " Nov 04 01:35:28.947946 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:28.947099 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-model-cache\") pod \"790de7b2-be85-47bb-b2f2-d21b1422bac9\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " Nov 04 01:35:28.947946 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:28.947168 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790de7b2-be85-47bb-b2f2-d21b1422bac9-kube-api-access-hsz4m\" (UniqueName: \"kubernetes.io/projected/790de7b2-be85-47bb-b2f2-d21b1422bac9-kube-api-access-hsz4m\") pod \"790de7b2-be85-47bb-b2f2-d21b1422bac9\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " Nov 04 01:35:28.947946 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:28.947202 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/790de7b2-be85-47bb-b2f2-d21b1422bac9-tls-certs\" (UniqueName: \"kubernetes.io/secret/790de7b2-be85-47bb-b2f2-d21b1422bac9-tls-certs\") pod \"790de7b2-be85-47bb-b2f2-d21b1422bac9\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " Nov 04 01:35:28.947946 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:28.947236 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-dshm\") pod \"790de7b2-be85-47bb-b2f2-d21b1422bac9\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " Nov 04 01:35:28.947946 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:28.947320 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-home\" (UniqueName: \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-home\") pod \"790de7b2-be85-47bb-b2f2-d21b1422bac9\" (UID: \"790de7b2-be85-47bb-b2f2-d21b1422bac9\") " Nov 04 01:35:28.948687 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:28.948463 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-home" pod "790de7b2-be85-47bb-b2f2-d21b1422bac9" (UID: "790de7b2-be85-47bb-b2f2-d21b1422bac9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:28.949037 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:28.948856 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-tmp-dir" pod "790de7b2-be85-47bb-b2f2-d21b1422bac9" (UID: "790de7b2-be85-47bb-b2f2-d21b1422bac9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:28.949452 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:28.949167 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-model-cache" pod "790de7b2-be85-47bb-b2f2-d21b1422bac9" (UID: "790de7b2-be85-47bb-b2f2-d21b1422bac9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:28.951959 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:28.951934 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-dshm" pod "790de7b2-be85-47bb-b2f2-d21b1422bac9" (UID: "790de7b2-be85-47bb-b2f2-d21b1422bac9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:28.953520 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:28.953497 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/790de7b2-be85-47bb-b2f2-d21b1422bac9-tls-certs" pod "790de7b2-be85-47bb-b2f2-d21b1422bac9" (UID: "790de7b2-be85-47bb-b2f2-d21b1422bac9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:28.956046 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:28.956012 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790de7b2-be85-47bb-b2f2-d21b1422bac9-kube-api-access-hsz4m" pod "790de7b2-be85-47bb-b2f2-d21b1422bac9" (UID: "790de7b2-be85-47bb-b2f2-d21b1422bac9"). InnerVolumeSpecName "kube-api-access-hsz4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:28.963274 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:28.963196 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:29.083766 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:29.079272 4852 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-tmp-dir\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:29.083766 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:29.079311 4852 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-model-cache\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:29.083766 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:29.079327 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsz4m\" (UniqueName: \"kubernetes.io/projected/790de7b2-be85-47bb-b2f2-d21b1422bac9-kube-api-access-hsz4m\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:29.083766 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:29.079340 4852 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/790de7b2-be85-47bb-b2f2-d21b1422bac9-tls-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:29.083766 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:29.079353 4852 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-dshm\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:29.083766 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:29.079367 4852 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/790de7b2-be85-47bb-b2f2-d21b1422bac9-home\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:29.092044 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:29.092010 4852 generic.go:356] "Generic (PLEG): container finished" podID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerID="e3260569ca48be59369e194750aa3a7ffc4a75f1c966ad60618289acd6c9e8f1" exitCode=0 Nov 04 01:35:29.092389 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:29.092363 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" event={"ID":"790de7b2-be85-47bb-b2f2-d21b1422bac9","Type":"ContainerDied","Data":"e3260569ca48be59369e194750aa3a7ffc4a75f1c966ad60618289acd6c9e8f1"} Nov 04 01:35:29.092598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:29.092580 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" event={"ID":"790de7b2-be85-47bb-b2f2-d21b1422bac9","Type":"ContainerDied","Data":"eebe9fe0df273524f1e51646195558b6aa46e75b7280de2fbd04d577a19debc2"} Nov 04 01:35:29.092773 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:29.092759 4852 scope.go:122] "RemoveContainer" containerID="e3260569ca48be59369e194750aa3a7ffc4a75f1c966ad60618289acd6c9e8f1" Nov 04 01:35:29.093063 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:29.093047 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" Nov 04 01:35:29.120470 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:29.120447 4852 scope.go:122] "RemoveContainer" containerID="e3260569ca48be59369e194750aa3a7ffc4a75f1c966ad60618289acd6c9e8f1" Nov 04 01:35:29.122106 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:35:29.122022 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3260569ca48be59369e194750aa3a7ffc4a75f1c966ad60618289acd6c9e8f1\": container with ID starting with e3260569ca48be59369e194750aa3a7ffc4a75f1c966ad60618289acd6c9e8f1 not found: ID does not exist" containerID="e3260569ca48be59369e194750aa3a7ffc4a75f1c966ad60618289acd6c9e8f1" Nov 04 01:35:29.122106 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:29.122061 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3260569ca48be59369e194750aa3a7ffc4a75f1c966ad60618289acd6c9e8f1"} err="failed to get container status \"e3260569ca48be59369e194750aa3a7ffc4a75f1c966ad60618289acd6c9e8f1\": rpc error: code = NotFound desc = could not find container \"e3260569ca48be59369e194750aa3a7ffc4a75f1c966ad60618289acd6c9e8f1\": container with ID starting with e3260569ca48be59369e194750aa3a7ffc4a75f1c966ad60618289acd6c9e8f1 not found: ID does not exist" Nov 04 01:35:29.135339 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:29.135224 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845"] Nov 04 01:35:29.158603 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:29.141642 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845"] Nov 04 01:35:29.388644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:29.383985 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:29.570938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:29.570908 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:29.801239 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:29.801191 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:29.814185 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:29.814141 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/premium-w6-499ddd-kserve-76554bcc7d-nz845" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 04 01:35:29.873207 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:35:29.873171 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:35:29.966602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:29.966545 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:29.997044 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:29.997008 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" path="/var/lib/kubelet/pods/790de7b2-be85-47bb-b2f2-d21b1422bac9/volumes" Nov 04 01:35:30.684535 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:30.684506 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:30.931989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:30.931952 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6"] Nov 04 01:35:30.932482 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:30.932191 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" podUID="baf9a879-211d-43e5-82ba-3909c2ee1b33" containerName="istio-proxy" containerID="cri-o://4ebd7897ca3f7299cf4740281616c06cb505ba9ab0442e8a1db672a9e98ced27" gracePeriod=30 Nov 04 01:35:30.954198 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:30.954105 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl"] Nov 04 01:35:30.963716 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:30.963691 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z"] Nov 04 01:35:30.965130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:30.964139 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" podUID="4bdbb81d-304f-4cb6-8282-813bf09b8699" containerName="istio-proxy" containerID="cri-o://53d99116f09a0c9f7dd551199583cc2094cb8b161cb4b19333c757d6712ab626" gracePeriod=30 Nov 04 01:35:30.965264 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:30.965228 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" podUID="26c2e335-58f4-4c3f-bea2-14a6dc9865ff" containerName="istio-proxy" containerID="cri-o://7d97b84216b7afaf499b293ba6fa8991b44e088053721eee46814e8ceec354fa" gracePeriod=30 Nov 04 01:35:30.969858 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:30.969802 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf"] Nov 04 01:35:30.969998 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:30.969973 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" podUID="8c157ec5-7fa3-4774-bace-71cd27ee9d82" containerName="istio-proxy" containerID="cri-o://cc4ef83ce21f3701151421380e82f68e3b9fffefcd9042a89312334a5ab74dcf" gracePeriod=30 Nov 04 01:35:31.076411 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:31.076365 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:31.271603 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:31.271556 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:31.615339 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:31.615246 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:35.094877 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:35.094845 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:35.891416 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:35.891388 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:36.133119 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.133081 4852 generic.go:356] "Generic (PLEG): container finished" podID="26c2e335-58f4-4c3f-bea2-14a6dc9865ff" containerID="7d97b84216b7afaf499b293ba6fa8991b44e088053721eee46814e8ceec354fa" exitCode=0 Nov 04 01:35:36.133586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.133153 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" event={"ID":"26c2e335-58f4-4c3f-bea2-14a6dc9865ff","Type":"ContainerDied","Data":"7d97b84216b7afaf499b293ba6fa8991b44e088053721eee46814e8ceec354fa"} Nov 04 01:35:36.142118 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.141904 4852 generic.go:356] "Generic (PLEG): container finished" podID="baf9a879-211d-43e5-82ba-3909c2ee1b33" containerID="4ebd7897ca3f7299cf4740281616c06cb505ba9ab0442e8a1db672a9e98ced27" exitCode=0 Nov 04 01:35:36.142118 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.142028 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" event={"ID":"baf9a879-211d-43e5-82ba-3909c2ee1b33","Type":"ContainerDied","Data":"4ebd7897ca3f7299cf4740281616c06cb505ba9ab0442e8a1db672a9e98ced27"} Nov 04 01:35:36.144643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.144444 4852 generic.go:356] "Generic (PLEG): container finished" podID="8c157ec5-7fa3-4774-bace-71cd27ee9d82" containerID="cc4ef83ce21f3701151421380e82f68e3b9fffefcd9042a89312334a5ab74dcf" exitCode=0 Nov 04 01:35:36.144643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.144537 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" event={"ID":"8c157ec5-7fa3-4774-bace-71cd27ee9d82","Type":"ContainerDied","Data":"cc4ef83ce21f3701151421380e82f68e3b9fffefcd9042a89312334a5ab74dcf"} Nov 04 01:35:36.152546 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.152517 4852 generic.go:356] "Generic (PLEG): container finished" podID="4bdbb81d-304f-4cb6-8282-813bf09b8699" containerID="53d99116f09a0c9f7dd551199583cc2094cb8b161cb4b19333c757d6712ab626" exitCode=0 Nov 04 01:35:36.152701 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.152667 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" event={"ID":"4bdbb81d-304f-4cb6-8282-813bf09b8699","Type":"ContainerDied","Data":"53d99116f09a0c9f7dd551199583cc2094cb8b161cb4b19333c757d6712ab626"} Nov 04 01:35:36.223054 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.223026 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:35:36.311345 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.311320 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:36.314627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.314606 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:35:36.317824 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.317808 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:36.362929 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.362906 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-podinfo\") pod \"baf9a879-211d-43e5-82ba-3909c2ee1b33\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " Nov 04 01:35:36.363057 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.362943 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-token\" (UniqueName: \"kubernetes.io/projected/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-token\") pod \"baf9a879-211d-43e5-82ba-3909c2ee1b33\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " Nov 04 01:35:36.363057 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.362969 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf9a879-211d-43e5-82ba-3909c2ee1b33-kube-api-access-hrmhw\" (UniqueName: \"kubernetes.io/projected/baf9a879-211d-43e5-82ba-3909c2ee1b33-kube-api-access-hrmhw\") pod \"baf9a879-211d-43e5-82ba-3909c2ee1b33\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " Nov 04 01:35:36.363057 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.363004 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baf9a879-211d-43e5-82ba-3909c2ee1b33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf9a879-211d-43e5-82ba-3909c2ee1b33-istiod-ca-cert\") pod \"baf9a879-211d-43e5-82ba-3909c2ee1b33\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " Nov 04 01:35:36.363057 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.363056 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-credential-socket\") pod \"baf9a879-211d-43e5-82ba-3909c2ee1b33\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " Nov 04 01:35:36.363275 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.363091 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-envoy\") pod \"baf9a879-211d-43e5-82ba-3909c2ee1b33\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " Nov 04 01:35:36.363275 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.363121 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-workload-certs\") pod \"baf9a879-211d-43e5-82ba-3909c2ee1b33\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " Nov 04 01:35:36.363275 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.363153 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-workload-socket\") pod \"baf9a879-211d-43e5-82ba-3909c2ee1b33\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " Nov 04 01:35:36.363437 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.363277 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-data\") pod \"baf9a879-211d-43e5-82ba-3909c2ee1b33\" (UID: \"baf9a879-211d-43e5-82ba-3909c2ee1b33\") " Nov 04 01:35:36.363825 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.363719 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-workload-certs" pod "baf9a879-211d-43e5-82ba-3909c2ee1b33" (UID: "baf9a879-211d-43e5-82ba-3909c2ee1b33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.363943 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.363796 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-workload-socket" pod "baf9a879-211d-43e5-82ba-3909c2ee1b33" (UID: "baf9a879-211d-43e5-82ba-3909c2ee1b33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.363943 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.363790 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-credential-socket" pod "baf9a879-211d-43e5-82ba-3909c2ee1b33" (UID: "baf9a879-211d-43e5-82ba-3909c2ee1b33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.364138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.364060 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-data" pod "baf9a879-211d-43e5-82ba-3909c2ee1b33" (UID: "baf9a879-211d-43e5-82ba-3909c2ee1b33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.364201 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.364156 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf9a879-211d-43e5-82ba-3909c2ee1b33-istiod-ca-cert" pod "baf9a879-211d-43e5-82ba-3909c2ee1b33" (UID: "baf9a879-211d-43e5-82ba-3909c2ee1b33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:36.365371 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.365351 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-podinfo" pod "baf9a879-211d-43e5-82ba-3909c2ee1b33" (UID: "baf9a879-211d-43e5-82ba-3909c2ee1b33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:36.365524 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.365507 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-token" pod "baf9a879-211d-43e5-82ba-3909c2ee1b33" (UID: "baf9a879-211d-43e5-82ba-3909c2ee1b33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.365854 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.365835 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-envoy" pod "baf9a879-211d-43e5-82ba-3909c2ee1b33" (UID: "baf9a879-211d-43e5-82ba-3909c2ee1b33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.365948 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.365934 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf9a879-211d-43e5-82ba-3909c2ee1b33-kube-api-access-hrmhw" pod "baf9a879-211d-43e5-82ba-3909c2ee1b33" (UID: "baf9a879-211d-43e5-82ba-3909c2ee1b33"). InnerVolumeSpecName "kube-api-access-hrmhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.464859 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.464781 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-credential-socket\") pod \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " Nov 04 01:35:36.464859 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.464824 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-envoy\") pod \"4bdbb81d-304f-4cb6-8282-813bf09b8699\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " Nov 04 01:35:36.465140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.464864 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-podinfo\") pod \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " Nov 04 01:35:36.465140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.464899 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istiod-ca-cert\") pod \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " Nov 04 01:35:36.465140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.464916 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-workload-socket\") pod \"4bdbb81d-304f-4cb6-8282-813bf09b8699\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " Nov 04 01:35:36.465140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.464941 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-workload-certs\") pod \"4bdbb81d-304f-4cb6-8282-813bf09b8699\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " Nov 04 01:35:36.465140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.464971 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c157ec5-7fa3-4774-bace-71cd27ee9d82-kube-api-access-f8wmd\" (UniqueName: \"kubernetes.io/projected/8c157ec5-7fa3-4774-bace-71cd27ee9d82-kube-api-access-f8wmd\") pod \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " Nov 04 01:35:36.465140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465015 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-credential-socket\") pod \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " Nov 04 01:35:36.465140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465044 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-podinfo\") pod \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " Nov 04 01:35:36.465140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465077 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-workload-socket\") pod \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " Nov 04 01:35:36.465140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465085 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-credential-socket" pod "26c2e335-58f4-4c3f-bea2-14a6dc9865ff" (UID: "26c2e335-58f4-4c3f-bea2-14a6dc9865ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.465140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465115 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-workload-socket\") pod \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " Nov 04 01:35:36.465140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465144 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-token\" (UniqueName: \"kubernetes.io/projected/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-token\") pod \"4bdbb81d-304f-4cb6-8282-813bf09b8699\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " Nov 04 01:35:36.465989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465177 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-credential-socket\") pod \"4bdbb81d-304f-4cb6-8282-813bf09b8699\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " Nov 04 01:35:36.465989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465205 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-workload-certs\") pod \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " Nov 04 01:35:36.465989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465234 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-data\") pod \"4bdbb81d-304f-4cb6-8282-813bf09b8699\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " Nov 04 01:35:36.465989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465275 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-envoy\") pod \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " Nov 04 01:35:36.465989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465307 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istiod-ca-cert\") pod \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " Nov 04 01:35:36.465989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465358 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-podinfo\") pod \"4bdbb81d-304f-4cb6-8282-813bf09b8699\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " Nov 04 01:35:36.465989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465395 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-kube-api-access-n4kdz\" (UniqueName: \"kubernetes.io/projected/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-kube-api-access-n4kdz\") pod \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " Nov 04 01:35:36.465989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465430 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-workload-certs\") pod \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " Nov 04 01:35:36.465989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465455 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-workload-socket" pod "26c2e335-58f4-4c3f-bea2-14a6dc9865ff" (UID: "26c2e335-58f4-4c3f-bea2-14a6dc9865ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.465989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465474 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bdbb81d-304f-4cb6-8282-813bf09b8699-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdbb81d-304f-4cb6-8282-813bf09b8699-istiod-ca-cert\") pod \"4bdbb81d-304f-4cb6-8282-813bf09b8699\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " Nov 04 01:35:36.465989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465507 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bdbb81d-304f-4cb6-8282-813bf09b8699-kube-api-access-gbp2n\" (UniqueName: \"kubernetes.io/projected/4bdbb81d-304f-4cb6-8282-813bf09b8699-kube-api-access-gbp2n\") pod \"4bdbb81d-304f-4cb6-8282-813bf09b8699\" (UID: \"4bdbb81d-304f-4cb6-8282-813bf09b8699\") " Nov 04 01:35:36.465989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465547 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-data\") pod \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " Nov 04 01:35:36.465989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465618 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-token\" (UniqueName: \"kubernetes.io/projected/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-token\") pod \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\" (UID: \"8c157ec5-7fa3-4774-bace-71cd27ee9d82\") " Nov 04 01:35:36.465989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465637 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-workload-certs" pod "4bdbb81d-304f-4cb6-8282-813bf09b8699" (UID: "4bdbb81d-304f-4cb6-8282-813bf09b8699"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.465989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465648 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-token\" (UniqueName: \"kubernetes.io/projected/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-token\") pod \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " Nov 04 01:35:36.465989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465686 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-envoy\") pod \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " Nov 04 01:35:36.465989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465726 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-data\") pod \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\" (UID: \"26c2e335-58f4-4c3f-bea2-14a6dc9865ff\") " Nov 04 01:35:36.465989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.465851 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-credential-socket" pod "4bdbb81d-304f-4cb6-8282-813bf09b8699" (UID: "4bdbb81d-304f-4cb6-8282-813bf09b8699"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.469167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.466022 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-workload-socket" pod "8c157ec5-7fa3-4774-bace-71cd27ee9d82" (UID: "8c157ec5-7fa3-4774-bace-71cd27ee9d82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.469167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.466272 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-workload-socket" pod "4bdbb81d-304f-4cb6-8282-813bf09b8699" (UID: "4bdbb81d-304f-4cb6-8282-813bf09b8699"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.469167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.466363 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istiod-ca-cert" pod "26c2e335-58f4-4c3f-bea2-14a6dc9865ff" (UID: "26c2e335-58f4-4c3f-bea2-14a6dc9865ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:36.469167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.466648 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-workload-certs" pod "26c2e335-58f4-4c3f-bea2-14a6dc9865ff" (UID: "26c2e335-58f4-4c3f-bea2-14a6dc9865ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.469167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.467260 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-data" pod "4bdbb81d-304f-4cb6-8282-813bf09b8699" (UID: "4bdbb81d-304f-4cb6-8282-813bf09b8699"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.469167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.467522 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-data" pod "26c2e335-58f4-4c3f-bea2-14a6dc9865ff" (UID: "26c2e335-58f4-4c3f-bea2-14a6dc9865ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.469167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.468272 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-data" pod "8c157ec5-7fa3-4774-bace-71cd27ee9d82" (UID: "8c157ec5-7fa3-4774-bace-71cd27ee9d82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.469167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.468522 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-workload-certs" pod "8c157ec5-7fa3-4774-bace-71cd27ee9d82" (UID: "8c157ec5-7fa3-4774-bace-71cd27ee9d82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.469167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.468606 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-envoy" pod "4bdbb81d-304f-4cb6-8282-813bf09b8699" (UID: "4bdbb81d-304f-4cb6-8282-813bf09b8699"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.469167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.468881 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.469167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.468893 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istiod-ca-cert" pod "8c157ec5-7fa3-4774-bace-71cd27ee9d82" (UID: "8c157ec5-7fa3-4774-bace-71cd27ee9d82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:36.469167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.468905 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.469167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.468922 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.469167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.468937 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.469167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.468951 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.469167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.468962 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.469167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.468974 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.469167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.468989 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.469167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.469003 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.469167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.469017 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.470892 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.468973 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c157ec5-7fa3-4774-bace-71cd27ee9d82-kube-api-access-f8wmd" pod "8c157ec5-7fa3-4774-bace-71cd27ee9d82" (UID: "8c157ec5-7fa3-4774-bace-71cd27ee9d82"). InnerVolumeSpecName "kube-api-access-f8wmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.471551 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.468971 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-podinfo" pod "26c2e335-58f4-4c3f-bea2-14a6dc9865ff" (UID: "26c2e335-58f4-4c3f-bea2-14a6dc9865ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:36.471551 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.469281 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bdbb81d-304f-4cb6-8282-813bf09b8699-istiod-ca-cert" pod "4bdbb81d-304f-4cb6-8282-813bf09b8699" (UID: "4bdbb81d-304f-4cb6-8282-813bf09b8699"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:36.471551 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.469541 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-credential-socket" pod "8c157ec5-7fa3-4774-bace-71cd27ee9d82" (UID: "8c157ec5-7fa3-4774-bace-71cd27ee9d82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.471551 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.469911 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-token" pod "26c2e335-58f4-4c3f-bea2-14a6dc9865ff" (UID: "26c2e335-58f4-4c3f-bea2-14a6dc9865ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.471551 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.470363 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-envoy" pod "26c2e335-58f4-4c3f-bea2-14a6dc9865ff" (UID: "26c2e335-58f4-4c3f-bea2-14a6dc9865ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.471551 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.470491 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.471551 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.471338 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.471551 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.471358 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.471551 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.471372 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.471551 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.471386 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.471551 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.471400 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrmhw\" (UniqueName: \"kubernetes.io/projected/baf9a879-211d-43e5-82ba-3909c2ee1b33-kube-api-access-hrmhw\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.471551 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.471417 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf9a879-211d-43e5-82ba-3909c2ee1b33-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.472943 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.471430 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.472943 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.472907 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-token" pod "8c157ec5-7fa3-4774-bace-71cd27ee9d82" (UID: "8c157ec5-7fa3-4774-bace-71cd27ee9d82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.473097 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.472957 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.473097 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.472916 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-kube-api-access-n4kdz" pod "26c2e335-58f4-4c3f-bea2-14a6dc9865ff" (UID: "26c2e335-58f4-4c3f-bea2-14a6dc9865ff"). InnerVolumeSpecName "kube-api-access-n4kdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.473097 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.472985 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.473097 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.473002 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.473097 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.473017 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf9a879-211d-43e5-82ba-3909c2ee1b33-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.475412 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.475386 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-envoy" pod "8c157ec5-7fa3-4774-bace-71cd27ee9d82" (UID: "8c157ec5-7fa3-4774-bace-71cd27ee9d82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.475412 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.475394 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-podinfo" pod "4bdbb81d-304f-4cb6-8282-813bf09b8699" (UID: "4bdbb81d-304f-4cb6-8282-813bf09b8699"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:36.483172 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.475667 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-podinfo" pod "8c157ec5-7fa3-4774-bace-71cd27ee9d82" (UID: "8c157ec5-7fa3-4774-bace-71cd27ee9d82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:36.485716 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.485641 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-token" pod "4bdbb81d-304f-4cb6-8282-813bf09b8699" (UID: "4bdbb81d-304f-4cb6-8282-813bf09b8699"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.485716 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.485689 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bdbb81d-304f-4cb6-8282-813bf09b8699-kube-api-access-gbp2n" pod "4bdbb81d-304f-4cb6-8282-813bf09b8699" (UID: "4bdbb81d-304f-4cb6-8282-813bf09b8699"). InnerVolumeSpecName "kube-api-access-gbp2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.529657 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.529601 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" containerID="cri-o://a84d99d9111bb846c2b4fb4770ed555ea310ff064d855527fe0a749c07c05863" gracePeriod=45 Nov 04 01:35:36.538478 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.538437 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" containerID="cri-o://50ba918b5aee6eeaeb7d9edaa4b2a0ab21e55c7e08d7a4ce9d0e7143e39d8cd6" gracePeriod=45 Nov 04 01:35:36.556520 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.556460 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" containerID="cri-o://0f1b55ff68e6a9d0c515e53b44b1636b6a5ece6920f3f61d39da27f1924d43e8" gracePeriod=45 Nov 04 01:35:36.574290 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.574265 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.574397 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.574318 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4kdz\" (UniqueName: \"kubernetes.io/projected/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-kube-api-access-n4kdz\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.574397 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.574337 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdbb81d-304f-4cb6-8282-813bf09b8699-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.574397 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.574350 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbp2n\" (UniqueName: \"kubernetes.io/projected/4bdbb81d-304f-4cb6-8282-813bf09b8699-kube-api-access-gbp2n\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.574397 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.574364 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.574397 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.574376 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.574397 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.574388 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.574680 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.574401 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c2e335-58f4-4c3f-bea2-14a6dc9865ff-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.574680 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.574415 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8wmd\" (UniqueName: \"kubernetes.io/projected/8c157ec5-7fa3-4774-bace-71cd27ee9d82-kube-api-access-f8wmd\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.574680 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.574428 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.574680 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.574442 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.574680 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.574463 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bdbb81d-304f-4cb6-8282-813bf09b8699-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.574680 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.574476 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.574680 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.574490 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c157ec5-7fa3-4774-bace-71cd27ee9d82-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.822192 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.821983 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:35:36.890909 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.890882 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:35:36.904722 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.904701 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:35:36.982051 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.981972 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-home\" (UniqueName: \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-home\") pod \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " Nov 04 01:35:36.982051 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.982017 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-dshm\") pod \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " Nov 04 01:35:36.982051 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.982046 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-model-cache\") pod \"934d8fb0-d7fd-4091-9077-f8445780adf3\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " Nov 04 01:35:36.982315 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.982069 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-model-cache\") pod \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " Nov 04 01:35:36.982315 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.982104 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/934d8fb0-d7fd-4091-9077-f8445780adf3-kube-api-access-4fv5n\" (UniqueName: \"kubernetes.io/projected/934d8fb0-d7fd-4091-9077-f8445780adf3-kube-api-access-4fv5n\") pod \"934d8fb0-d7fd-4091-9077-f8445780adf3\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " Nov 04 01:35:36.982315 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.982124 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-tls-certs\") pod \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " Nov 04 01:35:36.982315 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.982153 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-model-cache\") pod \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " Nov 04 01:35:36.982315 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.982176 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-dshm\") pod \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " Nov 04 01:35:36.982315 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.982202 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecc4ef25-c12e-4b56-b505-1435c1e0822c-tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc4ef25-c12e-4b56-b505-1435c1e0822c-tls-certs\") pod \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " Nov 04 01:35:36.982315 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.982251 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-tmp-dir\") pod \"934d8fb0-d7fd-4091-9077-f8445780adf3\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " Nov 04 01:35:36.982315 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.982273 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc4ef25-c12e-4b56-b505-1435c1e0822c-kube-api-access-6zt8x\" (UniqueName: \"kubernetes.io/projected/ecc4ef25-c12e-4b56-b505-1435c1e0822c-kube-api-access-6zt8x\") pod \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " Nov 04 01:35:36.982315 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.982305 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-tmp-dir\") pod \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " Nov 04 01:35:36.982884 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.982338 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/934d8fb0-d7fd-4091-9077-f8445780adf3-tls-certs\" (UniqueName: \"kubernetes.io/secret/934d8fb0-d7fd-4091-9077-f8445780adf3-tls-certs\") pod \"934d8fb0-d7fd-4091-9077-f8445780adf3\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " Nov 04 01:35:36.982884 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.982369 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-kube-api-access-fz2tl\" (UniqueName: \"kubernetes.io/projected/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-kube-api-access-fz2tl\") pod \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\" (UID: \"3becce48-d40e-4ee9-9a2f-b553bec9f2c5\") " Nov 04 01:35:36.982884 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.982391 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-tmp-dir\") pod \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " Nov 04 01:35:36.982884 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.982415 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-home\" (UniqueName: \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-home\") pod \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\" (UID: \"ecc4ef25-c12e-4b56-b505-1435c1e0822c\") " Nov 04 01:35:36.982884 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.982431 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-home\" (UniqueName: \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-home\") pod \"934d8fb0-d7fd-4091-9077-f8445780adf3\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " Nov 04 01:35:36.982884 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.982452 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-dshm\") pod \"934d8fb0-d7fd-4091-9077-f8445780adf3\" (UID: \"934d8fb0-d7fd-4091-9077-f8445780adf3\") " Nov 04 01:35:36.983255 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.983227 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-model-cache" pod "3becce48-d40e-4ee9-9a2f-b553bec9f2c5" (UID: "3becce48-d40e-4ee9-9a2f-b553bec9f2c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.983404 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.983388 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-home" pod "3becce48-d40e-4ee9-9a2f-b553bec9f2c5" (UID: "3becce48-d40e-4ee9-9a2f-b553bec9f2c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.986538 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.985609 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-model-cache" pod "934d8fb0-d7fd-4091-9077-f8445780adf3" (UID: "934d8fb0-d7fd-4091-9077-f8445780adf3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.986538 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.985892 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-model-cache" pod "ecc4ef25-c12e-4b56-b505-1435c1e0822c" (UID: "ecc4ef25-c12e-4b56-b505-1435c1e0822c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.986538 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.986325 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-tmp-dir" pod "ecc4ef25-c12e-4b56-b505-1435c1e0822c" (UID: "ecc4ef25-c12e-4b56-b505-1435c1e0822c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.989149 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.987225 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-tmp-dir" pod "3becce48-d40e-4ee9-9a2f-b553bec9f2c5" (UID: "3becce48-d40e-4ee9-9a2f-b553bec9f2c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.989149 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.987285 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-tmp-dir" pod "934d8fb0-d7fd-4091-9077-f8445780adf3" (UID: "934d8fb0-d7fd-4091-9077-f8445780adf3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.989149 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.987464 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-home" pod "ecc4ef25-c12e-4b56-b505-1435c1e0822c" (UID: "ecc4ef25-c12e-4b56-b505-1435c1e0822c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.989149 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.989102 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-home" pod "934d8fb0-d7fd-4091-9077-f8445780adf3" (UID: "934d8fb0-d7fd-4091-9077-f8445780adf3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.991616 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.991574 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-dshm" pod "3becce48-d40e-4ee9-9a2f-b553bec9f2c5" (UID: "3becce48-d40e-4ee9-9a2f-b553bec9f2c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.993019 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.992994 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-tls-certs" pod "3becce48-d40e-4ee9-9a2f-b553bec9f2c5" (UID: "3becce48-d40e-4ee9-9a2f-b553bec9f2c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:36.995317 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.995290 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-kube-api-access-fz2tl" pod "3becce48-d40e-4ee9-9a2f-b553bec9f2c5" (UID: "3becce48-d40e-4ee9-9a2f-b553bec9f2c5"). InnerVolumeSpecName "kube-api-access-fz2tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.995420 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.995308 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecc4ef25-c12e-4b56-b505-1435c1e0822c-tls-certs" pod "ecc4ef25-c12e-4b56-b505-1435c1e0822c" (UID: "ecc4ef25-c12e-4b56-b505-1435c1e0822c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:36.996885 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.996860 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/934d8fb0-d7fd-4091-9077-f8445780adf3-tls-certs" pod "934d8fb0-d7fd-4091-9077-f8445780adf3" (UID: "934d8fb0-d7fd-4091-9077-f8445780adf3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:36.997011 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.996993 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-dshm" pod "934d8fb0-d7fd-4091-9077-f8445780adf3" (UID: "934d8fb0-d7fd-4091-9077-f8445780adf3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.997074 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.997025 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc4ef25-c12e-4b56-b505-1435c1e0822c-kube-api-access-6zt8x" pod "ecc4ef25-c12e-4b56-b505-1435c1e0822c" (UID: "ecc4ef25-c12e-4b56-b505-1435c1e0822c"). InnerVolumeSpecName "kube-api-access-6zt8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.998282 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.998261 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-dshm" pod "ecc4ef25-c12e-4b56-b505-1435c1e0822c" (UID: "ecc4ef25-c12e-4b56-b505-1435c1e0822c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.998800 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:36.998780 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934d8fb0-d7fd-4091-9077-f8445780adf3-kube-api-access-4fv5n" pod "934d8fb0-d7fd-4091-9077-f8445780adf3" (UID: "934d8fb0-d7fd-4091-9077-f8445780adf3"). InnerVolumeSpecName "kube-api-access-4fv5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:37.084006 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.083974 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz2tl\" (UniqueName: \"kubernetes.io/projected/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-kube-api-access-fz2tl\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.084006 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.084005 4852 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-tmp-dir\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.084173 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.084015 4852 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-home\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.084173 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.084023 4852 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-home\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.084173 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.084031 4852 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-dshm\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.084173 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.084040 4852 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-home\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.084173 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.084047 4852 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-dshm\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.084173 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.084055 4852 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-model-cache\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.084173 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.084064 4852 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-model-cache\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.084173 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.084073 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fv5n\" (UniqueName: \"kubernetes.io/projected/934d8fb0-d7fd-4091-9077-f8445780adf3-kube-api-access-4fv5n\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.084173 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.084082 4852 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-tls-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.084173 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.084090 4852 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-model-cache\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.084173 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.084098 4852 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc4ef25-c12e-4b56-b505-1435c1e0822c-dshm\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.084173 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.084107 4852 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc4ef25-c12e-4b56-b505-1435c1e0822c-tls-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.084173 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.084115 4852 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/934d8fb0-d7fd-4091-9077-f8445780adf3-tmp-dir\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.084173 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.084123 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zt8x\" (UniqueName: \"kubernetes.io/projected/ecc4ef25-c12e-4b56-b505-1435c1e0822c-kube-api-access-6zt8x\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.084173 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.084131 4852 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3becce48-d40e-4ee9-9a2f-b553bec9f2c5-tmp-dir\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.084173 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.084139 4852 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/934d8fb0-d7fd-4091-9077-f8445780adf3-tls-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.160368 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.160337 4852 generic.go:356] "Generic (PLEG): container finished" podID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerID="50ba918b5aee6eeaeb7d9edaa4b2a0ab21e55c7e08d7a4ce9d0e7143e39d8cd6" exitCode=0 Nov 04 01:35:37.160839 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.160399 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" Nov 04 01:35:37.160839 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.160425 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" event={"ID":"934d8fb0-d7fd-4091-9077-f8445780adf3","Type":"ContainerDied","Data":"50ba918b5aee6eeaeb7d9edaa4b2a0ab21e55c7e08d7a4ce9d0e7143e39d8cd6"} Nov 04 01:35:37.160839 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.160481 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" event={"ID":"934d8fb0-d7fd-4091-9077-f8445780adf3","Type":"ContainerDied","Data":"075647edf31c04b840f30986fd9c9e22f57881092b7d0faaead97d9b58cd0913"} Nov 04 01:35:37.160839 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.160506 4852 scope.go:122] "RemoveContainer" containerID="50ba918b5aee6eeaeb7d9edaa4b2a0ab21e55c7e08d7a4ce9d0e7143e39d8cd6" Nov 04 01:35:37.162245 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.162220 4852 generic.go:356] "Generic (PLEG): container finished" podID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerID="a84d99d9111bb846c2b4fb4770ed555ea310ff064d855527fe0a749c07c05863" exitCode=0 Nov 04 01:35:37.162357 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.162269 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" Nov 04 01:35:37.162357 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.162311 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" event={"ID":"3becce48-d40e-4ee9-9a2f-b553bec9f2c5","Type":"ContainerDied","Data":"a84d99d9111bb846c2b4fb4770ed555ea310ff064d855527fe0a749c07c05863"} Nov 04 01:35:37.162648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.162362 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p" event={"ID":"3becce48-d40e-4ee9-9a2f-b553bec9f2c5","Type":"ContainerDied","Data":"437952ee9c8d56bf4b2dbb938f203cab9dcef3d7f8e7552842f629e56dfefd55"} Nov 04 01:35:37.164523 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.164144 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" Nov 04 01:35:37.164523 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.164152 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z" event={"ID":"26c2e335-58f4-4c3f-bea2-14a6dc9865ff","Type":"ContainerDied","Data":"dd7e3c1b0c9cae8eaad35de506dc47b18e89552ffebf39c02e2ac64e04b21f91"} Nov 04 01:35:37.173096 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.173072 4852 scope.go:122] "RemoveContainer" containerID="50ba918b5aee6eeaeb7d9edaa4b2a0ab21e55c7e08d7a4ce9d0e7143e39d8cd6" Nov 04 01:35:37.173434 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:35:37.173405 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50ba918b5aee6eeaeb7d9edaa4b2a0ab21e55c7e08d7a4ce9d0e7143e39d8cd6\": container with ID starting with 50ba918b5aee6eeaeb7d9edaa4b2a0ab21e55c7e08d7a4ce9d0e7143e39d8cd6 not found: ID does not exist" containerID="50ba918b5aee6eeaeb7d9edaa4b2a0ab21e55c7e08d7a4ce9d0e7143e39d8cd6" Nov 04 01:35:37.173552 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.173444 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50ba918b5aee6eeaeb7d9edaa4b2a0ab21e55c7e08d7a4ce9d0e7143e39d8cd6"} err="failed to get container status \"50ba918b5aee6eeaeb7d9edaa4b2a0ab21e55c7e08d7a4ce9d0e7143e39d8cd6\": rpc error: code = NotFound desc = could not find container \"50ba918b5aee6eeaeb7d9edaa4b2a0ab21e55c7e08d7a4ce9d0e7143e39d8cd6\": container with ID starting with 50ba918b5aee6eeaeb7d9edaa4b2a0ab21e55c7e08d7a4ce9d0e7143e39d8cd6 not found: ID does not exist" Nov 04 01:35:37.173552 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.173467 4852 scope.go:122] "RemoveContainer" containerID="a84d99d9111bb846c2b4fb4770ed555ea310ff064d855527fe0a749c07c05863" Nov 04 01:35:37.175057 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.175040 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" Nov 04 01:35:37.175288 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.175044 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6" event={"ID":"baf9a879-211d-43e5-82ba-3909c2ee1b33","Type":"ContainerDied","Data":"9bba0632d5739a86f6de6b261cfab07bbb41789f411eba0740e9f9ca49814477"} Nov 04 01:35:37.177337 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.177312 4852 generic.go:356] "Generic (PLEG): container finished" podID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerID="0f1b55ff68e6a9d0c515e53b44b1636b6a5ece6920f3f61d39da27f1924d43e8" exitCode=0 Nov 04 01:35:37.177431 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.177346 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" Nov 04 01:35:37.177431 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.177391 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" event={"ID":"ecc4ef25-c12e-4b56-b505-1435c1e0822c","Type":"ContainerDied","Data":"0f1b55ff68e6a9d0c515e53b44b1636b6a5ece6920f3f61d39da27f1924d43e8"} Nov 04 01:35:37.177613 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.177428 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb" event={"ID":"ecc4ef25-c12e-4b56-b505-1435c1e0822c","Type":"ContainerDied","Data":"005424728e7b934d24e694c55f6c21d14432b523cac734489fd2b17ff6951a04"} Nov 04 01:35:37.179154 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.179136 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" event={"ID":"8c157ec5-7fa3-4774-bace-71cd27ee9d82","Type":"ContainerDied","Data":"ebc7b1771464d3567a87837fe18c475e0c6359b9398dd6f632d21412af77eb6b"} Nov 04 01:35:37.179276 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.179141 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf" Nov 04 01:35:37.181411 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.181386 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" event={"ID":"4bdbb81d-304f-4cb6-8282-813bf09b8699","Type":"ContainerDied","Data":"e6665f8c8440b1f2078d728b6b836d9aeede193bb9e733550ff7e55610117fe2"} Nov 04 01:35:37.181520 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.181498 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl" Nov 04 01:35:37.191731 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.191715 4852 scope.go:122] "RemoveContainer" containerID="a84d99d9111bb846c2b4fb4770ed555ea310ff064d855527fe0a749c07c05863" Nov 04 01:35:37.192089 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:35:37.192070 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a84d99d9111bb846c2b4fb4770ed555ea310ff064d855527fe0a749c07c05863\": container with ID starting with a84d99d9111bb846c2b4fb4770ed555ea310ff064d855527fe0a749c07c05863 not found: ID does not exist" containerID="a84d99d9111bb846c2b4fb4770ed555ea310ff064d855527fe0a749c07c05863" Nov 04 01:35:37.192177 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.192094 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a84d99d9111bb846c2b4fb4770ed555ea310ff064d855527fe0a749c07c05863"} err="failed to get container status \"a84d99d9111bb846c2b4fb4770ed555ea310ff064d855527fe0a749c07c05863\": rpc error: code = NotFound desc = could not find container \"a84d99d9111bb846c2b4fb4770ed555ea310ff064d855527fe0a749c07c05863\": container with ID starting with a84d99d9111bb846c2b4fb4770ed555ea310ff064d855527fe0a749c07c05863 not found: ID does not exist" Nov 04 01:35:37.192177 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.192110 4852 scope.go:122] "RemoveContainer" containerID="7d97b84216b7afaf499b293ba6fa8991b44e088053721eee46814e8ceec354fa" Nov 04 01:35:37.201768 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.201743 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p"] Nov 04 01:35:37.203533 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.203513 4852 scope.go:122] "RemoveContainer" containerID="4ebd7897ca3f7299cf4740281616c06cb505ba9ab0442e8a1db672a9e98ced27" Nov 04 01:35:37.205542 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.205522 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/distinct-w2-d7e45d-kserve-7664dd8886-xs76p"] Nov 04 01:35:37.213336 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.213316 4852 scope.go:122] "RemoveContainer" containerID="0f1b55ff68e6a9d0c515e53b44b1636b6a5ece6920f3f61d39da27f1924d43e8" Nov 04 01:35:37.221130 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.221106 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6"] Nov 04 01:35:37.226160 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.225003 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6"] Nov 04 01:35:37.228766 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.228748 4852 scope.go:122] "RemoveContainer" containerID="0f1b55ff68e6a9d0c515e53b44b1636b6a5ece6920f3f61d39da27f1924d43e8" Nov 04 01:35:37.229072 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:35:37.229051 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f1b55ff68e6a9d0c515e53b44b1636b6a5ece6920f3f61d39da27f1924d43e8\": container with ID starting with 0f1b55ff68e6a9d0c515e53b44b1636b6a5ece6920f3f61d39da27f1924d43e8 not found: ID does not exist" containerID="0f1b55ff68e6a9d0c515e53b44b1636b6a5ece6920f3f61d39da27f1924d43e8" Nov 04 01:35:37.229181 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.229077 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f1b55ff68e6a9d0c515e53b44b1636b6a5ece6920f3f61d39da27f1924d43e8"} err="failed to get container status \"0f1b55ff68e6a9d0c515e53b44b1636b6a5ece6920f3f61d39da27f1924d43e8\": rpc error: code = NotFound desc = could not find container \"0f1b55ff68e6a9d0c515e53b44b1636b6a5ece6920f3f61d39da27f1924d43e8\": container with ID starting with 0f1b55ff68e6a9d0c515e53b44b1636b6a5ece6920f3f61d39da27f1924d43e8 not found: ID does not exist" Nov 04 01:35:37.229181 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.229097 4852 scope.go:122] "RemoveContainer" containerID="cc4ef83ce21f3701151421380e82f68e3b9fffefcd9042a89312334a5ab74dcf" Nov 04 01:35:37.239844 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.239759 4852 scope.go:122] "RemoveContainer" containerID="53d99116f09a0c9f7dd551199583cc2094cb8b161cb4b19333c757d6712ab626" Nov 04 01:35:37.245534 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.245487 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl"] Nov 04 01:35:37.247838 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.247818 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-b9dcl"] Nov 04 01:35:37.259661 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.259641 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb"] Nov 04 01:35:37.267764 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.267744 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/simulator-w2-d7e45d-kserve-7fb8bd674b-zpwbb"] Nov 04 01:35:37.281361 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.281344 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z"] Nov 04 01:35:37.287941 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.287921 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-chk5z"] Nov 04 01:35:37.305078 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.305058 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6"] Nov 04 01:35:37.320178 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.320153 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-blkc6"] Nov 04 01:35:37.332185 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.332163 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf"] Nov 04 01:35:37.342464 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.342436 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-rlnlf"] Nov 04 01:35:37.753904 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.753862 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/premium-w2-d7e45d-kserve-565dd7dbb7-wz7w6" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:37.994050 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.994008 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26c2e335-58f4-4c3f-bea2-14a6dc9865ff" path="/var/lib/kubelet/pods/26c2e335-58f4-4c3f-bea2-14a6dc9865ff/volumes" Nov 04 01:35:37.994482 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.994453 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" path="/var/lib/kubelet/pods/3becce48-d40e-4ee9-9a2f-b553bec9f2c5/volumes" Nov 04 01:35:37.994935 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.994916 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bdbb81d-304f-4cb6-8282-813bf09b8699" path="/var/lib/kubelet/pods/4bdbb81d-304f-4cb6-8282-813bf09b8699/volumes" Nov 04 01:35:37.995753 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.995726 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c157ec5-7fa3-4774-bace-71cd27ee9d82" path="/var/lib/kubelet/pods/8c157ec5-7fa3-4774-bace-71cd27ee9d82/volumes" Nov 04 01:35:37.996123 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.996106 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" path="/var/lib/kubelet/pods/934d8fb0-d7fd-4091-9077-f8445780adf3/volumes" Nov 04 01:35:37.996516 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.996500 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baf9a879-211d-43e5-82ba-3909c2ee1b33" path="/var/lib/kubelet/pods/baf9a879-211d-43e5-82ba-3909c2ee1b33/volumes" Nov 04 01:35:37.997167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:37.997147 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" path="/var/lib/kubelet/pods/ecc4ef25-c12e-4b56-b505-1435c1e0822c/volumes" Nov 04 01:35:38.262154 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:38.262127 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:38.628180 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:38.628084 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:38.794242 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:38.794191 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:39.521405 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:39.521378 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:39.785697 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:39.785668 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:40.925755 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:40.925722 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:41.121767 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:41.121729 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:44.199941 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.199907 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk"] Nov 04 01:35:44.200309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200217 4852 cpu_manager.go:414] "containerMap: removing container" podUID="b0515d69-ac7d-4d43-a316-58ccc63b757a" containerName="istio-proxy" Nov 04 01:35:44.200309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200229 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0515d69-ac7d-4d43-a316-58ccc63b757a" containerName="istio-proxy" Nov 04 01:35:44.200309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200244 4852 cpu_manager.go:414] "containerMap: removing container" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" Nov 04 01:35:44.200309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200249 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" Nov 04 01:35:44.200309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200260 4852 cpu_manager.go:414] "containerMap: removing container" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" Nov 04 01:35:44.200309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200265 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" Nov 04 01:35:44.200309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200272 4852 cpu_manager.go:414] "containerMap: removing container" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" Nov 04 01:35:44.200309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200276 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" Nov 04 01:35:44.200309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200285 4852 cpu_manager.go:414] "containerMap: removing container" podUID="b0515d69-ac7d-4d43-a316-58ccc63b757a" containerName="istio-proxy" Nov 04 01:35:44.200309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200290 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0515d69-ac7d-4d43-a316-58ccc63b757a" containerName="istio-proxy" Nov 04 01:35:44.200309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200300 4852 cpu_manager.go:414] "containerMap: removing container" podUID="46120899-8e38-4afd-8e30-787092a30fc8" containerName="revoke-keys" Nov 04 01:35:44.200309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200304 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46120899-8e38-4afd-8e30-787092a30fc8" containerName="revoke-keys" Nov 04 01:35:44.200309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200311 4852 cpu_manager.go:414] "containerMap: removing container" podUID="baf9a879-211d-43e5-82ba-3909c2ee1b33" containerName="istio-proxy" Nov 04 01:35:44.200309 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200315 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf9a879-211d-43e5-82ba-3909c2ee1b33" containerName="istio-proxy" Nov 04 01:35:44.200786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200324 4852 cpu_manager.go:414] "containerMap: removing container" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" Nov 04 01:35:44.200786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200328 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" Nov 04 01:35:44.200786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200334 4852 cpu_manager.go:414] "containerMap: removing container" podUID="26c2e335-58f4-4c3f-bea2-14a6dc9865ff" containerName="istio-proxy" Nov 04 01:35:44.200786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200338 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26c2e335-58f4-4c3f-bea2-14a6dc9865ff" containerName="istio-proxy" Nov 04 01:35:44.200786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200347 4852 cpu_manager.go:414] "containerMap: removing container" podUID="8c157ec5-7fa3-4774-bace-71cd27ee9d82" containerName="istio-proxy" Nov 04 01:35:44.200786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200351 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c157ec5-7fa3-4774-bace-71cd27ee9d82" containerName="istio-proxy" Nov 04 01:35:44.200786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200357 4852 cpu_manager.go:414] "containerMap: removing container" podUID="4bdbb81d-304f-4cb6-8282-813bf09b8699" containerName="istio-proxy" Nov 04 01:35:44.200786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200361 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bdbb81d-304f-4cb6-8282-813bf09b8699" containerName="istio-proxy" Nov 04 01:35:44.200786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200413 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c157ec5-7fa3-4774-bace-71cd27ee9d82" containerName="istio-proxy" Nov 04 01:35:44.200786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200420 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bdbb81d-304f-4cb6-8282-813bf09b8699" containerName="istio-proxy" Nov 04 01:35:44.200786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200428 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0515d69-ac7d-4d43-a316-58ccc63b757a" containerName="istio-proxy" Nov 04 01:35:44.200786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200435 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="790de7b2-be85-47bb-b2f2-d21b1422bac9" containerName="main" Nov 04 01:35:44.200786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200440 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="934d8fb0-d7fd-4091-9077-f8445780adf3" containerName="main" Nov 04 01:35:44.200786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200446 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="3becce48-d40e-4ee9-9a2f-b553bec9f2c5" containerName="main" Nov 04 01:35:44.200786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200453 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="46120899-8e38-4afd-8e30-787092a30fc8" containerName="revoke-keys" Nov 04 01:35:44.200786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200458 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf9a879-211d-43e5-82ba-3909c2ee1b33" containerName="istio-proxy" Nov 04 01:35:44.200786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200463 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc4ef25-c12e-4b56-b505-1435c1e0822c" containerName="main" Nov 04 01:35:44.200786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.200467 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="26c2e335-58f4-4c3f-bea2-14a6dc9865ff" containerName="istio-proxy" Nov 04 01:35:44.207029 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.207010 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.219596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.219554 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk"] Nov 04 01:35:44.221064 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.221042 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7"] Nov 04 01:35:44.221701 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.221663 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0515d69-ac7d-4d43-a316-58ccc63b757a" containerName="istio-proxy" Nov 04 01:35:44.225255 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.225232 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.237785 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.237763 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7"] Nov 04 01:35:44.242959 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.242887 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc"] Nov 04 01:35:44.247625 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.247608 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.264268 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.260692 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc"] Nov 04 01:35:44.350479 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.350438 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.350479 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.350479 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.350709 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.350498 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d1708da-d20c-474b-81d6-7eac1175396d-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.350709 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.350594 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zrzm\" (UniqueName: \"kubernetes.io/projected/6d1708da-d20c-474b-81d6-7eac1175396d-kube-api-access-9zrzm\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.350709 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.350659 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.350709 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.350694 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.350881 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.350727 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.350881 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.350751 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.350881 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.350835 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.350881 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.350860 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d1708da-d20c-474b-81d6-7eac1175396d-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.350881 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.350874 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d1708da-d20c-474b-81d6-7eac1175396d-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.351119 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.350918 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.351119 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.351007 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.351119 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.351035 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.351119 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.351072 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.351295 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.351120 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.351295 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.351163 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c583de-adfc-4b67-8b77-eeddbc699916-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.351295 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.351183 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.351295 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.351209 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.351295 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.351230 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.351510 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.351318 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.351510 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.351381 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.351510 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.351452 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.351510 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.351470 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.351510 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.351488 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.351706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.351527 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz4p5\" (UniqueName: \"kubernetes.io/projected/f2c583de-adfc-4b67-8b77-eeddbc699916-kube-api-access-zz4p5\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.351706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.351551 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnc95\" (UniqueName: \"kubernetes.io/projected/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-kube-api-access-pnc95\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.431908 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.431883 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7"] Nov 04 01:35:44.435275 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.435261 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.449808 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.449783 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7"] Nov 04 01:35:44.452237 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.452171 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.452534 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.452506 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.452638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.452553 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz4p5\" (UniqueName: \"kubernetes.io/projected/f2c583de-adfc-4b67-8b77-eeddbc699916-kube-api-access-zz4p5\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.452638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.452596 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnc95\" (UniqueName: \"kubernetes.io/projected/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-kube-api-access-pnc95\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.452638 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.452622 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.452800 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.452654 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.452800 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.452677 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d1708da-d20c-474b-81d6-7eac1175396d-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.452800 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.452710 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zrzm\" (UniqueName: \"kubernetes.io/projected/6d1708da-d20c-474b-81d6-7eac1175396d-kube-api-access-9zrzm\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.452800 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.452768 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.452800 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.452793 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.453017 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.452824 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.453017 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.452847 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.453017 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.452881 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.453017 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.452904 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d1708da-d20c-474b-81d6-7eac1175396d-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.453017 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.452943 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d1708da-d20c-474b-81d6-7eac1175396d-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.453017 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.452982 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.454140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.453025 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.454140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.453050 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.454140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.453093 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.454140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.453138 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.454140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.453161 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c583de-adfc-4b67-8b77-eeddbc699916-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.454140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.453189 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.454140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.453220 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.454140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.453277 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.454140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.453307 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.454140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.453380 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.454140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.453415 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.454760 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.454185 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.454760 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.454486 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.455931 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.455909 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.456062 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.455923 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw"] Nov 04 01:35:44.456683 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.456638 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.457139 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.457110 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.457283 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.457258 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.458453 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.457515 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.458453 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.457656 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d1708da-d20c-474b-81d6-7eac1175396d-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.458453 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.457911 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.458453 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.458128 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.458453 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.458157 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.458453 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.456096 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.458453 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.458427 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.459116 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.458509 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.459116 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.458943 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.459116 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.458948 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c583de-adfc-4b67-8b77-eeddbc699916-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.459521 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.459500 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.460122 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.460100 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.460701 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.460682 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.460936 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.460912 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d1708da-d20c-474b-81d6-7eac1175396d-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.461648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.461628 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.461899 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.461878 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.469165 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.469147 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt"] Nov 04 01:35:44.473910 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.473526 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.474045 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.474010 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw"] Nov 04 01:35:44.474792 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.474772 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz4p5\" (UniqueName: \"kubernetes.io/projected/f2c583de-adfc-4b67-8b77-eeddbc699916-kube-api-access-zz4p5\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.478521 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.478458 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d1708da-d20c-474b-81d6-7eac1175396d-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.478823 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.478798 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.482215 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.482175 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zrzm\" (UniqueName: \"kubernetes.io/projected/6d1708da-d20c-474b-81d6-7eac1175396d-kube-api-access-9zrzm\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.483168 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.483145 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnc95\" (UniqueName: \"kubernetes.io/projected/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-kube-api-access-pnc95\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.484698 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.484674 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.485575 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.485537 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:35:44.498270 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.498246 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt"] Nov 04 01:35:44.521334 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.521314 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:44.538265 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.538241 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:44.554992 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.554961 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/897931d4-4332-4739-8bd8-93433e6611be-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.555129 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.555000 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/897931d4-4332-4739-8bd8-93433e6611be-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.555129 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.555023 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.555246 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.555127 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.555246 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.555152 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.555246 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.555169 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.555246 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.555187 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.555423 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.555320 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwzgv\" (UniqueName: \"kubernetes.io/projected/897931d4-4332-4739-8bd8-93433e6611be-kube-api-access-zwzgv\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.555423 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.555346 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.555423 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.555363 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.555597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.555417 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.555597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.555494 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc3454a1-295f-441f-9e75-54d0dc18bfea-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.555597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.555517 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/897931d4-4332-4739-8bd8-93433e6611be-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.555597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.555535 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.555597 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.555594 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.555868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.555663 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj4fp\" (UniqueName: \"kubernetes.io/projected/fc3454a1-295f-441f-9e75-54d0dc18bfea-kube-api-access-wj4fp\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.555868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.555726 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.555868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.555753 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.570001 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.569759 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.656742 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.656832 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/897931d4-4332-4739-8bd8-93433e6611be-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.656866 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/897931d4-4332-4739-8bd8-93433e6611be-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.656905 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.656932 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.656957 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26phm\" (UniqueName: \"kubernetes.io/projected/1daad035-b4ae-4b3c-b522-cc267c4bf053-kube-api-access-26phm\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.657008 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.657033 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.657060 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.657082 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.657128 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.657180 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1daad035-b4ae-4b3c-b522-cc267c4bf053-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.657217 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwzgv\" (UniqueName: \"kubernetes.io/projected/897931d4-4332-4739-8bd8-93433e6611be-kube-api-access-zwzgv\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.657242 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.657258 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.657275 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.657291 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.657318 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc3454a1-295f-441f-9e75-54d0dc18bfea-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.657340 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/897931d4-4332-4739-8bd8-93433e6611be-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.657358 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.657382 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.657405 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.657421 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.657436 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.657460 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj4fp\" (UniqueName: \"kubernetes.io/projected/fc3454a1-295f-441f-9e75-54d0dc18bfea-kube-api-access-wj4fp\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.657485 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.657501 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.663209 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.663437 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.664138 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.664704 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/897931d4-4332-4739-8bd8-93433e6611be-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.664939 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.665557 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc3454a1-295f-441f-9e75-54d0dc18bfea-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.667521 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwzgv\" (UniqueName: \"kubernetes.io/projected/897931d4-4332-4739-8bd8-93433e6611be-kube-api-access-zwzgv\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.668380 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.668595 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.669033 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.669239 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.671706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.669680 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.676046 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.675969 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/897931d4-4332-4739-8bd8-93433e6611be-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.677924 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.677110 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj4fp\" (UniqueName: \"kubernetes.io/projected/fc3454a1-295f-441f-9e75-54d0dc18bfea-kube-api-access-wj4fp\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.677924 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.677547 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/897931d4-4332-4739-8bd8-93433e6611be-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.677924 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.677888 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.681661 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.681620 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.681867 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.681842 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.751109 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.750733 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:44.759387 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.759355 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.759511 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.759449 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.759511 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.759485 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.759652 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.759525 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.759652 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.759595 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.759752 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.759699 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.759752 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.759745 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26phm\" (UniqueName: \"kubernetes.io/projected/1daad035-b4ae-4b3c-b522-cc267c4bf053-kube-api-access-26phm\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.759870 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.759795 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.759870 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.759857 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1daad035-b4ae-4b3c-b522-cc267c4bf053-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.762547 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.760853 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.762547 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.761459 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.762547 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.761731 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.762547 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.761805 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.763019 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.762543 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1daad035-b4ae-4b3c-b522-cc267c4bf053-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.777956 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.776210 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.777956 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.777515 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.778330 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.778306 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.780747 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.780699 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26phm\" (UniqueName: \"kubernetes.io/projected/1daad035-b4ae-4b3c-b522-cc267c4bf053-kube-api-access-26phm\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.815674 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.815638 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:44.825156 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.824789 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:44.884612 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.874696 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk"] Nov 04 01:35:44.934387 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.934313 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7"] Nov 04 01:35:44.939914 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.939851 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc"] Nov 04 01:35:44.954825 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.953782 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:44.954825 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.953869 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:44.954825 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.953914 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:44.954825 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.954524 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:44.954825 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.954623 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:44.954825 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.954666 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:44.963009 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.961398 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:44.963009 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.961455 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:44.963009 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:44.961506 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:45.086070 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:45.085992 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7"] Nov 04 01:35:45.128159 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:35:45.128126 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3454a1_295f_441f_9e75_54d0dc18bfea.slice/crio-c39973b11b77639dbce887555b38939a3cf7ff05e325ad426669a53a98c096a5 WatchSource:0}: Error finding container c39973b11b77639dbce887555b38939a3cf7ff05e325ad426669a53a98c096a5: Status 404 returned error can't find the container with id c39973b11b77639dbce887555b38939a3cf7ff05e325ad426669a53a98c096a5 Nov 04 01:35:45.142651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:45.142238 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:45.142651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:45.142319 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:45.142651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:45.142359 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:45.231135 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:45.231055 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" event={"ID":"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b","Type":"ContainerStarted","Data":"7c6dbd9c11ead259e396052c654fc791fe0e9ea717250f5f4888512bae6e30bc"} Nov 04 01:35:45.237453 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:45.237164 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" event={"ID":"6d1708da-d20c-474b-81d6-7eac1175396d","Type":"ContainerStarted","Data":"c24110ab1d61194572045742b3f0c50d224f5a99e72412e8a7c47d9969e0a48a"} Nov 04 01:35:45.239782 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:45.239121 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" event={"ID":"f2c583de-adfc-4b67-8b77-eeddbc699916","Type":"ContainerStarted","Data":"a7f90696f9ddea15ad4ebdb703a96e1acd464bc5d01917cdbbd545af2caaabfe"} Nov 04 01:35:45.239782 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:45.239165 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" event={"ID":"f2c583de-adfc-4b67-8b77-eeddbc699916","Type":"ContainerStarted","Data":"3d74140eb92760a5407faa2ea6efd822504b956ae84578f12ae11753b3d23eee"} Nov 04 01:35:45.241821 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:45.241762 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" event={"ID":"fc3454a1-295f-441f-9e75-54d0dc18bfea","Type":"ContainerStarted","Data":"c39973b11b77639dbce887555b38939a3cf7ff05e325ad426669a53a98c096a5"} Nov 04 01:35:45.256698 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:45.256676 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw"] Nov 04 01:35:45.284751 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:45.284022 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:45.284751 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:45.284123 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:45.284751 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:45.284167 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:45.316271 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:45.315129 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" podStartSLOduration=1.3151105379999999 podStartE2EDuration="1.315110538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:45.275452862 +0000 UTC m=+2227.390562981" watchObservedRunningTime="2026-11-04 01:35:45.315110538 +0000 UTC m=+2227.430220656" Nov 04 01:35:45.316271 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:45.316183 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt"] Nov 04 01:35:45.352655 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:45.352618 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:45.353149 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:45.353124 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:45.353300 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:45.353283 4852 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:45.540641 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:45.538920 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:46.260973 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.259927 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" event={"ID":"897931d4-4332-4739-8bd8-93433e6611be","Type":"ContainerStarted","Data":"b87cf4bd9ba0921b91d73becb244925c229605d58aa51a0f99f3cff4dc530e31"} Nov 04 01:35:46.260973 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.260011 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" event={"ID":"897931d4-4332-4739-8bd8-93433e6611be","Type":"ContainerStarted","Data":"14189b7354b84dde9c5ee51fb5d482033c493d196ba7130b2701de4f26152c2b"} Nov 04 01:35:46.266521 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.265667 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" event={"ID":"fc3454a1-295f-441f-9e75-54d0dc18bfea","Type":"ContainerStarted","Data":"84dffa77ab7ca1db3a1a9561f28a83d77af02e281ee4cac3463e443e6a417331"} Nov 04 01:35:46.269074 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.269027 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" event={"ID":"1daad035-b4ae-4b3c-b522-cc267c4bf053","Type":"ContainerStarted","Data":"6419fe3a1faff4bcf7ecb634f2b37cacc7f3682c8663a1bdf53bfc7ed03d0c33"} Nov 04 01:35:46.269342 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.269306 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" event={"ID":"1daad035-b4ae-4b3c-b522-cc267c4bf053","Type":"ContainerStarted","Data":"ccd90436ba78c02ac1b4194da83529b098fe741e2de4862e90383ecb46f3c66a"} Nov 04 01:35:46.279595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.273843 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" event={"ID":"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b","Type":"ContainerStarted","Data":"46f421afab4c386d8ac7b9a0c9c5aa29282813cc0ca661457fcfe617426062f6"} Nov 04 01:35:46.279595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.278213 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" event={"ID":"6d1708da-d20c-474b-81d6-7eac1175396d","Type":"ContainerStarted","Data":"4d79f9417d7eba574eddb05fc04af12ac054aac9b059649e823fe78420c1d9f0"} Nov 04 01:35:46.286718 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.286625 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" podStartSLOduration=2.286604028 podStartE2EDuration="2.286604028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:46.282071489 +0000 UTC m=+2228.397181608" watchObservedRunningTime="2026-11-04 01:35:46.286604028 +0000 UTC m=+2228.401714148" Nov 04 01:35:46.341146 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.341062 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" podStartSLOduration=2.341044181 podStartE2EDuration="2.341044181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:46.340872071 +0000 UTC m=+2228.455982190" watchObservedRunningTime="2026-11-04 01:35:46.341044181 +0000 UTC m=+2228.456154291" Nov 04 01:35:46.343301 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.343249 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" podStartSLOduration=2.343231403 podStartE2EDuration="2.343231403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:46.307677651 +0000 UTC m=+2228.422787772" watchObservedRunningTime="2026-11-04 01:35:46.343231403 +0000 UTC m=+2228.458341522" Nov 04 01:35:46.367587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.367416 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" podStartSLOduration=2.367396748 podStartE2EDuration="2.367396748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:46.366846358 +0000 UTC m=+2228.481956477" watchObservedRunningTime="2026-11-04 01:35:46.367396748 +0000 UTC m=+2228.482506869" Nov 04 01:35:46.389588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.388076 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" podStartSLOduration=2.38805593 podStartE2EDuration="2.38805593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:46.387091123 +0000 UTC m=+2228.502201244" watchObservedRunningTime="2026-11-04 01:35:46.38805593 +0000 UTC m=+2228.503166050" Nov 04 01:35:46.533446 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.533339 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:46.533801 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.533778 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 04 01:35:46.533942 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.533918 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" podUID="6d1708da-d20c-474b-81d6-7eac1175396d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 04 01:35:46.543796 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.543760 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 04 01:35:46.544080 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.544052 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" podUID="f2c583de-adfc-4b67-8b77-eeddbc699916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 04 01:35:46.578588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.578131 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:46.582587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.580203 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 04 01:35:46.582587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.580256 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" podUID="cd9afd3b-9e83-4731-9b32-305bc5e4eb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 04 01:35:46.697587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.697276 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:46.764075 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.764031 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:46.766511 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.766483 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 04 01:35:46.767220 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.766845 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" podUID="fc3454a1-295f-441f-9e75-54d0dc18bfea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 04 01:35:46.819893 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.819791 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:46.822156 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.822113 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 04 01:35:46.822395 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.822370 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" podUID="897931d4-4332-4739-8bd8-93433e6611be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 04 01:35:46.836814 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.836773 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:46.837203 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.837166 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 04 01:35:46.837377 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.837354 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" podUID="1daad035-b4ae-4b3c-b522-cc267c4bf053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 04 01:35:46.870888 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:46.870840 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:47.545590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:47.544692 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 04 01:35:47.545590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:47.544783 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" podUID="6d1708da-d20c-474b-81d6-7eac1175396d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 04 01:35:47.545590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:47.544888 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 04 01:35:47.545590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:47.544950 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" podUID="f2c583de-adfc-4b67-8b77-eeddbc699916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 04 01:35:47.584117 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:47.584081 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 04 01:35:47.584947 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:47.584913 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" podUID="cd9afd3b-9e83-4731-9b32-305bc5e4eb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 04 01:35:47.757607 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:47.755046 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 04 01:35:47.757607 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:47.755114 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" podUID="fc3454a1-295f-441f-9e75-54d0dc18bfea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 04 01:35:47.817651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:47.817155 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 04 01:35:47.817651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:47.817235 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" podUID="897931d4-4332-4739-8bd8-93433e6611be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 04 01:35:47.826895 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:47.826850 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 04 01:35:47.827144 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:47.827118 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" podUID="1daad035-b4ae-4b3c-b522-cc267c4bf053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 04 01:35:48.526590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:48.526160 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 04 01:35:48.526590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:48.526234 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" podUID="6d1708da-d20c-474b-81d6-7eac1175396d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 04 01:35:48.543085 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:48.543051 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 04 01:35:48.543360 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:48.543325 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" podUID="f2c583de-adfc-4b67-8b77-eeddbc699916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 04 01:35:48.571053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:48.571020 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 04 01:35:48.571746 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:48.571716 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" podUID="cd9afd3b-9e83-4731-9b32-305bc5e4eb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 04 01:35:48.753501 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:48.753042 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 04 01:35:48.753501 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:48.753107 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" podUID="fc3454a1-295f-441f-9e75-54d0dc18bfea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 04 01:35:48.819221 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:48.819121 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 04 01:35:48.819488 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:48.819446 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" podUID="897931d4-4332-4739-8bd8-93433e6611be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 04 01:35:48.826066 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:48.826026 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 04 01:35:48.826267 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:48.826243 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" podUID="1daad035-b4ae-4b3c-b522-cc267c4bf053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 04 01:35:49.522077 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:49.522037 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 04 01:35:49.522362 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:49.522330 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" podUID="6d1708da-d20c-474b-81d6-7eac1175396d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 04 01:35:49.538990 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:49.538959 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 04 01:35:49.539202 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:49.539175 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" podUID="f2c583de-adfc-4b67-8b77-eeddbc699916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 04 01:35:49.571084 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:49.571050 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 04 01:35:49.571717 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:49.571665 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" podUID="cd9afd3b-9e83-4731-9b32-305bc5e4eb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 04 01:35:49.752076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:49.752037 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 04 01:35:49.752356 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:49.752328 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" podUID="fc3454a1-295f-441f-9e75-54d0dc18bfea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 04 01:35:49.817158 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:49.817059 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 04 01:35:49.817418 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:49.817369 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" podUID="897931d4-4332-4739-8bd8-93433e6611be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 04 01:35:49.826774 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:49.826747 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 04 01:35:49.826985 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:49.826961 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" podUID="1daad035-b4ae-4b3c-b522-cc267c4bf053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 04 01:35:50.099101 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:50.098975 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:50.260075 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:50.260039 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:50.528588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:50.525957 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 04 01:35:50.528588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:50.526020 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" podUID="6d1708da-d20c-474b-81d6-7eac1175396d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 04 01:35:50.545341 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:50.545302 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 04 01:35:50.545656 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:50.545624 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" podUID="f2c583de-adfc-4b67-8b77-eeddbc699916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 04 01:35:50.573959 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:50.573920 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 04 01:35:50.574525 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:50.574491 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" podUID="cd9afd3b-9e83-4731-9b32-305bc5e4eb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 04 01:35:50.765996 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:50.765957 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 04 01:35:50.766242 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:50.766215 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" podUID="fc3454a1-295f-441f-9e75-54d0dc18bfea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 04 01:35:50.823144 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:50.823048 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 04 01:35:50.823421 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:50.823370 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" podUID="897931d4-4332-4739-8bd8-93433e6611be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 04 01:35:50.835459 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:50.835429 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 04 01:35:50.835729 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:50.835691 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" podUID="1daad035-b4ae-4b3c-b522-cc267c4bf053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 04 01:35:51.522585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:51.522001 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 04 01:35:51.522585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:51.522067 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" podUID="6d1708da-d20c-474b-81d6-7eac1175396d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 04 01:35:51.540589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:51.538923 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 04 01:35:51.540589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:51.538983 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" podUID="f2c583de-adfc-4b67-8b77-eeddbc699916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 04 01:35:51.570957 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:51.570922 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 04 01:35:51.571232 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:51.571203 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" podUID="cd9afd3b-9e83-4731-9b32-305bc5e4eb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 04 01:35:51.755996 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:51.755956 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 04 01:35:51.756644 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:51.756608 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" podUID="fc3454a1-295f-441f-9e75-54d0dc18bfea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 04 01:35:51.829649 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:51.828933 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 04 01:35:51.829649 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:51.828996 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" podUID="897931d4-4332-4739-8bd8-93433e6611be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 04 01:35:51.829933 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:51.829653 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 04 01:35:51.829933 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:51.829697 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" podUID="1daad035-b4ae-4b3c-b522-cc267c4bf053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 04 01:35:52.522590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:52.521961 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 04 01:35:52.522590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:52.522028 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" podUID="6d1708da-d20c-474b-81d6-7eac1175396d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 04 01:35:52.542709 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:52.542672 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 04 01:35:52.543002 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:52.542961 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" podUID="f2c583de-adfc-4b67-8b77-eeddbc699916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 04 01:35:52.571131 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:52.571000 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 04 01:35:52.571131 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:52.571062 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" podUID="cd9afd3b-9e83-4731-9b32-305bc5e4eb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 04 01:35:52.752435 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:52.751996 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 04 01:35:52.752435 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:52.752056 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" podUID="fc3454a1-295f-441f-9e75-54d0dc18bfea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 04 01:35:52.818070 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:52.817973 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 04 01:35:52.818754 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:52.818706 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" podUID="897931d4-4332-4739-8bd8-93433e6611be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 04 01:35:52.829579 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:52.829536 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 04 01:35:52.829764 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:52.829739 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" podUID="1daad035-b4ae-4b3c-b522-cc267c4bf053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 04 01:35:53.133347 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:53.133255 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:53.466663 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:53.442735 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:53.517592 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:53.514034 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc"] Nov 04 01:35:53.517912 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:53.517890 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk"] Nov 04 01:35:53.518041 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:53.518029 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb"] Nov 04 01:35:53.521588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:53.518306 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" podUID="c0958a62-b79b-4d8f-9a10-81acf1522245" containerName="istio-proxy" containerID="cri-o://be1b10acb88d6fe373bfa9a321293554c5d732bed733b08d4d26cd9ffc3e5a93" gracePeriod=30 Nov 04 01:35:53.521849 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:53.520203 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" podUID="6d1708da-d20c-474b-81d6-7eac1175396d" containerName="istio-proxy" containerID="cri-o://4d79f9417d7eba574eddb05fc04af12ac054aac9b059649e823fe78420c1d9f0" gracePeriod=30 Nov 04 01:35:53.522062 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:53.521607 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" podUID="cd9afd3b-9e83-4731-9b32-305bc5e4eb5b" containerName="istio-proxy" containerID="cri-o://46f421afab4c386d8ac7b9a0c9c5aa29282813cc0ca661457fcfe617426062f6" gracePeriod=30 Nov 04 01:35:53.525699 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:53.522950 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7"] Nov 04 01:35:53.585020 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:53.584975 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" podUID="f2c583de-adfc-4b67-8b77-eeddbc699916" containerName="istio-proxy" containerID="cri-o://a7f90696f9ddea15ad4ebdb703a96e1acd464bc5d01917cdbbd545af2caaabfe" gracePeriod=30 Nov 04 01:35:53.598345 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:53.598318 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:53.825589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:53.800731 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 04 01:35:53.825589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:53.800812 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" podUID="fc3454a1-295f-441f-9e75-54d0dc18bfea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 04 01:35:53.825589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:53.818133 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 04 01:35:53.825589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:53.818189 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" podUID="897931d4-4332-4739-8bd8-93433e6611be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 04 01:35:53.855814 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:53.855778 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 04 01:35:53.856112 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:53.856068 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" podUID="1daad035-b4ae-4b3c-b522-cc267c4bf053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 04 01:35:54.751982 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:54.751943 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 04 01:35:54.752269 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:54.752238 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" podUID="fc3454a1-295f-441f-9e75-54d0dc18bfea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 04 01:35:54.818099 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:54.818065 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 04 01:35:54.818405 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:54.818355 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" podUID="897931d4-4332-4739-8bd8-93433e6611be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 04 01:35:54.825673 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:54.825648 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 04 01:35:54.826166 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:54.826140 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" podUID="1daad035-b4ae-4b3c-b522-cc267c4bf053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 04 01:35:55.752476 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:55.752014 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 04 01:35:55.752476 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:55.752093 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" podUID="fc3454a1-295f-441f-9e75-54d0dc18bfea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 04 01:35:55.819080 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:55.819045 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 04 01:35:55.819363 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:55.819317 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" podUID="897931d4-4332-4739-8bd8-93433e6611be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 04 01:35:55.828231 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:55.828203 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 04 01:35:55.828866 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:55.828828 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" podUID="1daad035-b4ae-4b3c-b522-cc267c4bf053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 04 01:35:56.751799 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:56.751759 4852 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 04 01:35:56.751961 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:56.751832 4852 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" podUID="fc3454a1-295f-441f-9e75-54d0dc18bfea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 04 01:35:56.862811 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:56.862775 4852 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:35:56.863422 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:56.863392 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" podUID="c0958a62-b79b-4d8f-9a10-81acf1522245" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:35:56.998666 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:56.997633 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:56.998666 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:56.998005 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:56.999045 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:56.999022 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:35:57.010578 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:57.010531 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:57.011537 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:57.011374 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:57.012664 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:57.012643 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:35:57.756526 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:57.756495 4852 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:57.757141 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:57.757119 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:57.757744 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:57.757730 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:35:57.816871 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:57.816820 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:57.982519 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:57.982473 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:58.346760 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.346727 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:58.547624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.547590 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:58.744912 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.744882 4852 generic.go:356] "Generic (PLEG): container finished" podID="c0958a62-b79b-4d8f-9a10-81acf1522245" containerID="be1b10acb88d6fe373bfa9a321293554c5d732bed733b08d4d26cd9ffc3e5a93" exitCode=0 Nov 04 01:35:58.745075 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.744957 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" event={"ID":"c0958a62-b79b-4d8f-9a10-81acf1522245","Type":"ContainerDied","Data":"be1b10acb88d6fe373bfa9a321293554c5d732bed733b08d4d26cd9ffc3e5a93"} Nov 04 01:35:58.779528 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.779504 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:35:58.910423 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.910361 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-token\" (UniqueName: \"kubernetes.io/projected/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-token\") pod \"c0958a62-b79b-4d8f-9a10-81acf1522245\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " Nov 04 01:35:58.910548 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.910440 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0958a62-b79b-4d8f-9a10-81acf1522245-kube-api-access-dhc9p\" (UniqueName: \"kubernetes.io/projected/c0958a62-b79b-4d8f-9a10-81acf1522245-kube-api-access-dhc9p\") pod \"c0958a62-b79b-4d8f-9a10-81acf1522245\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " Nov 04 01:35:58.910548 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.910473 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-credential-socket\") pod \"c0958a62-b79b-4d8f-9a10-81acf1522245\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " Nov 04 01:35:58.910664 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.910626 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-workload-socket\") pod \"c0958a62-b79b-4d8f-9a10-81acf1522245\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " Nov 04 01:35:58.910717 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.910688 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-podinfo\") pod \"c0958a62-b79b-4d8f-9a10-81acf1522245\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " Nov 04 01:35:58.910767 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.910717 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0958a62-b79b-4d8f-9a10-81acf1522245-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0958a62-b79b-4d8f-9a10-81acf1522245-istiod-ca-cert\") pod \"c0958a62-b79b-4d8f-9a10-81acf1522245\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " Nov 04 01:35:58.910767 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.910762 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-envoy\") pod \"c0958a62-b79b-4d8f-9a10-81acf1522245\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " Nov 04 01:35:58.910872 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.910794 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-workload-certs\") pod \"c0958a62-b79b-4d8f-9a10-81acf1522245\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " Nov 04 01:35:58.910872 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.910788 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-credential-socket" pod "c0958a62-b79b-4d8f-9a10-81acf1522245" (UID: "c0958a62-b79b-4d8f-9a10-81acf1522245"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.910872 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.910820 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-data\") pod \"c0958a62-b79b-4d8f-9a10-81acf1522245\" (UID: \"c0958a62-b79b-4d8f-9a10-81acf1522245\") " Nov 04 01:35:58.910872 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.910850 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-workload-socket" pod "c0958a62-b79b-4d8f-9a10-81acf1522245" (UID: "c0958a62-b79b-4d8f-9a10-81acf1522245"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.911077 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.911060 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-data" pod "c0958a62-b79b-4d8f-9a10-81acf1522245" (UID: "c0958a62-b79b-4d8f-9a10-81acf1522245"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.911183 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.911091 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-workload-certs" pod "c0958a62-b79b-4d8f-9a10-81acf1522245" (UID: "c0958a62-b79b-4d8f-9a10-81acf1522245"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.911251 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.911234 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0958a62-b79b-4d8f-9a10-81acf1522245-istiod-ca-cert" pod "c0958a62-b79b-4d8f-9a10-81acf1522245" (UID: "c0958a62-b79b-4d8f-9a10-81acf1522245"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:58.911976 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.911945 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.912096 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.911981 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0958a62-b79b-4d8f-9a10-81acf1522245-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.912096 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.911996 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.912096 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.912012 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.912096 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.912028 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.913142 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.913114 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0958a62-b79b-4d8f-9a10-81acf1522245-kube-api-access-dhc9p" pod "c0958a62-b79b-4d8f-9a10-81acf1522245" (UID: "c0958a62-b79b-4d8f-9a10-81acf1522245"). InnerVolumeSpecName "kube-api-access-dhc9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:58.913251 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.913198 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-podinfo" pod "c0958a62-b79b-4d8f-9a10-81acf1522245" (UID: "c0958a62-b79b-4d8f-9a10-81acf1522245"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:58.913312 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.913280 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-token" pod "c0958a62-b79b-4d8f-9a10-81acf1522245" (UID: "c0958a62-b79b-4d8f-9a10-81acf1522245"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:58.913380 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:58.913310 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-envoy" pod "c0958a62-b79b-4d8f-9a10-81acf1522245" (UID: "c0958a62-b79b-4d8f-9a10-81acf1522245"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:59.013057 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:59.013025 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:59.013057 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:59.013054 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:59.013403 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:59.013068 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0958a62-b79b-4d8f-9a10-81acf1522245-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:59.013403 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:59.013082 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhc9p\" (UniqueName: \"kubernetes.io/projected/c0958a62-b79b-4d8f-9a10-81acf1522245-kube-api-access-dhc9p\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:59.521862 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:59.521830 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:35:59.539006 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:59.538980 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:35:59.570894 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:59.570868 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:35:59.669916 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:59.669891 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:35:59.750909 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:59.750872 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" event={"ID":"c0958a62-b79b-4d8f-9a10-81acf1522245","Type":"ContainerDied","Data":"d0c30ed8b7dc5991dede8152d98d98e43727ec37a270aefb776b647f5ae75fe0"} Nov 04 01:35:59.751037 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:59.750950 4852 scope.go:122] "RemoveContainer" containerID="be1b10acb88d6fe373bfa9a321293554c5d732bed733b08d4d26cd9ffc3e5a93" Nov 04 01:35:59.751037 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:59.750895 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb" Nov 04 01:35:59.773353 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:59.773303 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb"] Nov 04 01:35:59.776779 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:59.776757 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-vrmhb"] Nov 04 01:35:59.866326 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:35:59.866291 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:35:59.993170 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:35:59.993143 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0958a62-b79b-4d8f-9a10-81acf1522245" path="/var/lib/kubelet/pods/c0958a62-b79b-4d8f-9a10-81acf1522245/volumes" Nov 04 01:36:00.202608 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:00.201942 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:36:00.460354 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:00.460227 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7"] Nov 04 01:36:00.463168 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:00.462532 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f"] Nov 04 01:36:00.463168 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:00.462848 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" podUID="df08c3cb-cf91-4bec-a39d-c32e07739abf" containerName="istio-proxy" containerID="cri-o://2bc3fdbad492290277c5ab697666237aba16f9910ad6ae894f80497a5fa6615c" gracePeriod=30 Nov 04 01:36:00.465770 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:00.465744 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw"] Nov 04 01:36:00.465941 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:00.465919 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" podUID="897931d4-4332-4739-8bd8-93433e6611be" containerName="istio-proxy" containerID="cri-o://b87cf4bd9ba0921b91d73becb244925c229605d58aa51a0f99f3cff4dc530e31" gracePeriod=30 Nov 04 01:36:00.470817 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:00.470793 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt"] Nov 04 01:36:00.470980 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:00.470960 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" podUID="1daad035-b4ae-4b3c-b522-cc267c4bf053" containerName="istio-proxy" containerID="cri-o://6419fe3a1faff4bcf7ecb634f2b37cacc7f3682c8663a1bdf53bfc7ed03d0c33" gracePeriod=30 Nov 04 01:36:00.523584 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:00.523534 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:36:00.757405 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:00.757375 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" podUID="fc3454a1-295f-441f-9e75-54d0dc18bfea" containerName="istio-proxy" containerID="cri-o://84dffa77ab7ca1db3a1a9561f28a83d77af02e281ee4cac3463e443e6a417331" gracePeriod=30 Nov 04 01:36:01.682757 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.682723 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:36:01.718865 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.718808 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:36:01.740775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.740750 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:36:01.766077 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.766044 4852 generic.go:356] "Generic (PLEG): container finished" podID="cd9afd3b-9e83-4731-9b32-305bc5e4eb5b" containerID="46f421afab4c386d8ac7b9a0c9c5aa29282813cc0ca661457fcfe617426062f6" exitCode=0 Nov 04 01:36:01.766230 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.766130 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" event={"ID":"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b","Type":"ContainerDied","Data":"46f421afab4c386d8ac7b9a0c9c5aa29282813cc0ca661457fcfe617426062f6"} Nov 04 01:36:01.766291 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.766256 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" Nov 04 01:36:01.766513 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.766481 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc" event={"ID":"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b","Type":"ContainerDied","Data":"7c6dbd9c11ead259e396052c654fc791fe0e9ea717250f5f4888512bae6e30bc"} Nov 04 01:36:01.766613 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.766532 4852 scope.go:122] "RemoveContainer" containerID="46f421afab4c386d8ac7b9a0c9c5aa29282813cc0ca661457fcfe617426062f6" Nov 04 01:36:01.768655 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.768630 4852 generic.go:356] "Generic (PLEG): container finished" podID="6d1708da-d20c-474b-81d6-7eac1175396d" containerID="4d79f9417d7eba574eddb05fc04af12ac054aac9b059649e823fe78420c1d9f0" exitCode=0 Nov 04 01:36:01.768777 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.768724 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" event={"ID":"6d1708da-d20c-474b-81d6-7eac1175396d","Type":"ContainerDied","Data":"4d79f9417d7eba574eddb05fc04af12ac054aac9b059649e823fe78420c1d9f0"} Nov 04 01:36:01.768777 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.768741 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" Nov 04 01:36:01.768777 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.768763 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk" event={"ID":"6d1708da-d20c-474b-81d6-7eac1175396d","Type":"ContainerDied","Data":"c24110ab1d61194572045742b3f0c50d224f5a99e72412e8a7c47d9969e0a48a"} Nov 04 01:36:01.773102 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.773081 4852 generic.go:356] "Generic (PLEG): container finished" podID="f2c583de-adfc-4b67-8b77-eeddbc699916" containerID="a7f90696f9ddea15ad4ebdb703a96e1acd464bc5d01917cdbbd545af2caaabfe" exitCode=0 Nov 04 01:36:01.773212 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.773199 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" event={"ID":"f2c583de-adfc-4b67-8b77-eeddbc699916","Type":"ContainerDied","Data":"a7f90696f9ddea15ad4ebdb703a96e1acd464bc5d01917cdbbd545af2caaabfe"} Nov 04 01:36:01.779292 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.779250 4852 scope.go:122] "RemoveContainer" containerID="46f421afab4c386d8ac7b9a0c9c5aa29282813cc0ca661457fcfe617426062f6" Nov 04 01:36:01.780724 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:36:01.780699 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f421afab4c386d8ac7b9a0c9c5aa29282813cc0ca661457fcfe617426062f6\": container with ID starting with 46f421afab4c386d8ac7b9a0c9c5aa29282813cc0ca661457fcfe617426062f6 not found: ID does not exist" containerID="46f421afab4c386d8ac7b9a0c9c5aa29282813cc0ca661457fcfe617426062f6" Nov 04 01:36:01.780887 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.780861 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f421afab4c386d8ac7b9a0c9c5aa29282813cc0ca661457fcfe617426062f6"} err="failed to get container status \"46f421afab4c386d8ac7b9a0c9c5aa29282813cc0ca661457fcfe617426062f6\": rpc error: code = NotFound desc = could not find container \"46f421afab4c386d8ac7b9a0c9c5aa29282813cc0ca661457fcfe617426062f6\": container with ID starting with 46f421afab4c386d8ac7b9a0c9c5aa29282813cc0ca661457fcfe617426062f6 not found: ID does not exist" Nov 04 01:36:01.780991 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.780979 4852 scope.go:122] "RemoveContainer" containerID="4d79f9417d7eba574eddb05fc04af12ac054aac9b059649e823fe78420c1d9f0" Nov 04 01:36:01.801104 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.801052 4852 scope.go:122] "RemoveContainer" containerID="4d79f9417d7eba574eddb05fc04af12ac054aac9b059649e823fe78420c1d9f0" Nov 04 01:36:01.801506 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:36:01.801479 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d79f9417d7eba574eddb05fc04af12ac054aac9b059649e823fe78420c1d9f0\": container with ID starting with 4d79f9417d7eba574eddb05fc04af12ac054aac9b059649e823fe78420c1d9f0 not found: ID does not exist" containerID="4d79f9417d7eba574eddb05fc04af12ac054aac9b059649e823fe78420c1d9f0" Nov 04 01:36:01.801658 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.801517 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d79f9417d7eba574eddb05fc04af12ac054aac9b059649e823fe78420c1d9f0"} err="failed to get container status \"4d79f9417d7eba574eddb05fc04af12ac054aac9b059649e823fe78420c1d9f0\": rpc error: code = NotFound desc = could not find container \"4d79f9417d7eba574eddb05fc04af12ac054aac9b059649e823fe78420c1d9f0\": container with ID starting with 4d79f9417d7eba574eddb05fc04af12ac054aac9b059649e823fe78420c1d9f0 not found: ID does not exist" Nov 04 01:36:01.839345 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.839324 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:36:01.844084 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844065 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d1708da-d20c-474b-81d6-7eac1175396d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d1708da-d20c-474b-81d6-7eac1175396d-istiod-ca-cert\") pod \"6d1708da-d20c-474b-81d6-7eac1175396d\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " Nov 04 01:36:01.844166 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844124 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-credential-socket\") pod \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " Nov 04 01:36:01.844166 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844153 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-token\" (UniqueName: \"kubernetes.io/projected/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-token\") pod \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " Nov 04 01:36:01.844257 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844181 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d1708da-d20c-474b-81d6-7eac1175396d-kube-api-access-9zrzm\" (UniqueName: \"kubernetes.io/projected/6d1708da-d20c-474b-81d6-7eac1175396d-kube-api-access-9zrzm\") pod \"6d1708da-d20c-474b-81d6-7eac1175396d\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " Nov 04 01:36:01.844257 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844198 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-workload-socket\") pod \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " Nov 04 01:36:01.844257 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844243 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d1708da-d20c-474b-81d6-7eac1175396d-istio-token\" (UniqueName: \"kubernetes.io/projected/6d1708da-d20c-474b-81d6-7eac1175396d-istio-token\") pod \"6d1708da-d20c-474b-81d6-7eac1175396d\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " Nov 04 01:36:01.844413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844269 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-envoy\") pod \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " Nov 04 01:36:01.844413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844289 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istiod-ca-cert\") pod \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " Nov 04 01:36:01.844413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844313 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-credential-socket\") pod \"6d1708da-d20c-474b-81d6-7eac1175396d\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " Nov 04 01:36:01.844413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844331 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-workload-certs\") pod \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " Nov 04 01:36:01.844413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844355 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-istio-envoy\") pod \"6d1708da-d20c-474b-81d6-7eac1175396d\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " Nov 04 01:36:01.844413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844378 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-credential-socket" pod "cd9afd3b-9e83-4731-9b32-305bc5e4eb5b" (UID: "cd9afd3b-9e83-4731-9b32-305bc5e4eb5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:01.844413 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844406 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-podinfo\") pod \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " Nov 04 01:36:01.844783 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844422 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-workload-certs\") pod \"6d1708da-d20c-474b-81d6-7eac1175396d\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " Nov 04 01:36:01.844783 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844433 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d1708da-d20c-474b-81d6-7eac1175396d-istiod-ca-cert" pod "6d1708da-d20c-474b-81d6-7eac1175396d" (UID: "6d1708da-d20c-474b-81d6-7eac1175396d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:01.844783 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844448 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-kube-api-access-pnc95\" (UniqueName: \"kubernetes.io/projected/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-kube-api-access-pnc95\") pod \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " Nov 04 01:36:01.844783 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844477 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-istio-data\") pod \"6d1708da-d20c-474b-81d6-7eac1175396d\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " Nov 04 01:36:01.844783 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844518 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-workload-socket\") pod \"6d1708da-d20c-474b-81d6-7eac1175396d\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " Nov 04 01:36:01.844783 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844549 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d1708da-d20c-474b-81d6-7eac1175396d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d1708da-d20c-474b-81d6-7eac1175396d-istio-podinfo\") pod \"6d1708da-d20c-474b-81d6-7eac1175396d\" (UID: \"6d1708da-d20c-474b-81d6-7eac1175396d\") " Nov 04 01:36:01.844783 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844591 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-data\") pod \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\" (UID: \"cd9afd3b-9e83-4731-9b32-305bc5e4eb5b\") " Nov 04 01:36:01.844783 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844742 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istiod-ca-cert" pod "cd9afd3b-9e83-4731-9b32-305bc5e4eb5b" (UID: "cd9afd3b-9e83-4731-9b32-305bc5e4eb5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:01.845182 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844847 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-workload-certs" pod "6d1708da-d20c-474b-81d6-7eac1175396d" (UID: "6d1708da-d20c-474b-81d6-7eac1175396d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:01.845182 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.844952 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-data" pod "cd9afd3b-9e83-4731-9b32-305bc5e4eb5b" (UID: "cd9afd3b-9e83-4731-9b32-305bc5e4eb5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:01.845182 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.845055 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-credential-socket" pod "6d1708da-d20c-474b-81d6-7eac1175396d" (UID: "6d1708da-d20c-474b-81d6-7eac1175396d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:01.845182 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.845143 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-istio-data" pod "6d1708da-d20c-474b-81d6-7eac1175396d" (UID: "6d1708da-d20c-474b-81d6-7eac1175396d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:01.845394 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.845259 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-workload-certs" pod "cd9afd3b-9e83-4731-9b32-305bc5e4eb5b" (UID: "cd9afd3b-9e83-4731-9b32-305bc5e4eb5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:01.845394 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.845334 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.845394 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.845379 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d1708da-d20c-474b-81d6-7eac1175396d-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.845588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.845397 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.845588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.845412 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.845588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.845427 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.845588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.845441 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.845588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.845455 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.845588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.845469 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.846056 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.845688 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-workload-socket" pod "6d1708da-d20c-474b-81d6-7eac1175396d" (UID: "6d1708da-d20c-474b-81d6-7eac1175396d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:01.847196 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.847170 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d1708da-d20c-474b-81d6-7eac1175396d-kube-api-access-9zrzm" pod "6d1708da-d20c-474b-81d6-7eac1175396d" (UID: "6d1708da-d20c-474b-81d6-7eac1175396d"). InnerVolumeSpecName "kube-api-access-9zrzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:01.847483 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.847460 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-podinfo" pod "cd9afd3b-9e83-4731-9b32-305bc5e4eb5b" (UID: "cd9afd3b-9e83-4731-9b32-305bc5e4eb5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:36:01.847574 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.847482 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-token" pod "cd9afd3b-9e83-4731-9b32-305bc5e4eb5b" (UID: "cd9afd3b-9e83-4731-9b32-305bc5e4eb5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:01.847750 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.847711 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-workload-socket" pod "cd9afd3b-9e83-4731-9b32-305bc5e4eb5b" (UID: "cd9afd3b-9e83-4731-9b32-305bc5e4eb5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:01.847826 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.847755 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-kube-api-access-pnc95" pod "cd9afd3b-9e83-4731-9b32-305bc5e4eb5b" (UID: "cd9afd3b-9e83-4731-9b32-305bc5e4eb5b"). InnerVolumeSpecName "kube-api-access-pnc95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:01.847826 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.847793 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-istio-envoy" pod "6d1708da-d20c-474b-81d6-7eac1175396d" (UID: "6d1708da-d20c-474b-81d6-7eac1175396d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:01.847924 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.847916 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-envoy" pod "cd9afd3b-9e83-4731-9b32-305bc5e4eb5b" (UID: "cd9afd3b-9e83-4731-9b32-305bc5e4eb5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:01.848148 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.848135 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d1708da-d20c-474b-81d6-7eac1175396d-istio-podinfo" pod "6d1708da-d20c-474b-81d6-7eac1175396d" (UID: "6d1708da-d20c-474b-81d6-7eac1175396d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:36:01.848332 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.848319 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d1708da-d20c-474b-81d6-7eac1175396d-istio-token" pod "6d1708da-d20c-474b-81d6-7eac1175396d" (UID: "6d1708da-d20c-474b-81d6-7eac1175396d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:01.946375 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.946349 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-credential-socket\") pod \"f2c583de-adfc-4b67-8b77-eeddbc699916\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " Nov 04 01:36:01.946535 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.946397 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c583de-adfc-4b67-8b77-eeddbc699916-kube-api-access-zz4p5\" (UniqueName: \"kubernetes.io/projected/f2c583de-adfc-4b67-8b77-eeddbc699916-kube-api-access-zz4p5\") pod \"f2c583de-adfc-4b67-8b77-eeddbc699916\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " Nov 04 01:36:01.946535 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.946440 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-data\") pod \"f2c583de-adfc-4b67-8b77-eeddbc699916\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " Nov 04 01:36:01.946535 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.946495 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-workload-certs\") pod \"f2c583de-adfc-4b67-8b77-eeddbc699916\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " Nov 04 01:36:01.946746 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.946621 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-envoy\") pod \"f2c583de-adfc-4b67-8b77-eeddbc699916\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " Nov 04 01:36:01.946746 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.946670 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-token\" (UniqueName: \"kubernetes.io/projected/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-token\") pod \"f2c583de-adfc-4b67-8b77-eeddbc699916\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " Nov 04 01:36:01.946746 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.946680 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-credential-socket" pod "f2c583de-adfc-4b67-8b77-eeddbc699916" (UID: "f2c583de-adfc-4b67-8b77-eeddbc699916"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:01.946746 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.946713 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-podinfo\") pod \"f2c583de-adfc-4b67-8b77-eeddbc699916\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " Nov 04 01:36:01.946958 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.946759 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-workload-socket\") pod \"f2c583de-adfc-4b67-8b77-eeddbc699916\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " Nov 04 01:36:01.946958 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.946781 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-workload-certs" pod "f2c583de-adfc-4b67-8b77-eeddbc699916" (UID: "f2c583de-adfc-4b67-8b77-eeddbc699916"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:01.946958 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.946794 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-data" pod "f2c583de-adfc-4b67-8b77-eeddbc699916" (UID: "f2c583de-adfc-4b67-8b77-eeddbc699916"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:01.946958 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.946790 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c583de-adfc-4b67-8b77-eeddbc699916-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c583de-adfc-4b67-8b77-eeddbc699916-istiod-ca-cert\") pod \"f2c583de-adfc-4b67-8b77-eeddbc699916\" (UID: \"f2c583de-adfc-4b67-8b77-eeddbc699916\") " Nov 04 01:36:01.947165 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.947036 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-workload-socket" pod "f2c583de-adfc-4b67-8b77-eeddbc699916" (UID: "f2c583de-adfc-4b67-8b77-eeddbc699916"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:01.947165 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.947109 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c583de-adfc-4b67-8b77-eeddbc699916-istiod-ca-cert" pod "f2c583de-adfc-4b67-8b77-eeddbc699916" (UID: "f2c583de-adfc-4b67-8b77-eeddbc699916"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:01.947437 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.947372 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnc95\" (UniqueName: \"kubernetes.io/projected/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-kube-api-access-pnc95\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.947437 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.947391 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.947437 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.947406 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c583de-adfc-4b67-8b77-eeddbc699916-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.947437 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.947420 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.947437 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.947435 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d1708da-d20c-474b-81d6-7eac1175396d-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.947727 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.947448 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.947727 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.947462 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.947727 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.947477 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zrzm\" (UniqueName: \"kubernetes.io/projected/6d1708da-d20c-474b-81d6-7eac1175396d-kube-api-access-9zrzm\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.947727 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.947491 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.947727 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.947504 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d1708da-d20c-474b-81d6-7eac1175396d-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.947727 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.947517 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.947727 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.947530 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.947727 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.947544 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d1708da-d20c-474b-81d6-7eac1175396d-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.947727 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.947557 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.947727 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.947604 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.948770 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.948743 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-envoy" pod "f2c583de-adfc-4b67-8b77-eeddbc699916" (UID: "f2c583de-adfc-4b67-8b77-eeddbc699916"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:01.948850 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.948829 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c583de-adfc-4b67-8b77-eeddbc699916-kube-api-access-zz4p5" pod "f2c583de-adfc-4b67-8b77-eeddbc699916" (UID: "f2c583de-adfc-4b67-8b77-eeddbc699916"). InnerVolumeSpecName "kube-api-access-zz4p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:01.948937 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.948853 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-podinfo" pod "f2c583de-adfc-4b67-8b77-eeddbc699916" (UID: "f2c583de-adfc-4b67-8b77-eeddbc699916"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:36:01.948972 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:01.948963 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-token" pod "f2c583de-adfc-4b67-8b77-eeddbc699916" (UID: "f2c583de-adfc-4b67-8b77-eeddbc699916"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:02.048622 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:02.048601 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz4p5\" (UniqueName: \"kubernetes.io/projected/f2c583de-adfc-4b67-8b77-eeddbc699916-kube-api-access-zz4p5\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:02.048622 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:02.048623 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:02.048775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:02.048633 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:02.048775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:02.048642 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c583de-adfc-4b67-8b77-eeddbc699916-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:02.083451 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:02.083421 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc"] Nov 04 01:36:02.088408 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:02.088387 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-r9qvc"] Nov 04 01:36:02.099657 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:02.099639 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk"] Nov 04 01:36:02.103989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:02.103967 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-pl4pk"] Nov 04 01:36:02.780475 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:02.780449 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" Nov 04 01:36:02.780896 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:02.780449 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7" event={"ID":"f2c583de-adfc-4b67-8b77-eeddbc699916","Type":"ContainerDied","Data":"3d74140eb92760a5407faa2ea6efd822504b956ae84578f12ae11753b3d23eee"} Nov 04 01:36:02.780896 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:02.780601 4852 scope.go:122] "RemoveContainer" containerID="a7f90696f9ddea15ad4ebdb703a96e1acd464bc5d01917cdbbd545af2caaabfe" Nov 04 01:36:02.797472 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:02.797449 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7"] Nov 04 01:36:02.801816 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:02.801790 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-4xxv7"] Nov 04 01:36:03.992713 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:03.992674 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d1708da-d20c-474b-81d6-7eac1175396d" path="/var/lib/kubelet/pods/6d1708da-d20c-474b-81d6-7eac1175396d/volumes" Nov 04 01:36:03.993283 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:03.993263 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd9afd3b-9e83-4731-9b32-305bc5e4eb5b" path="/var/lib/kubelet/pods/cd9afd3b-9e83-4731-9b32-305bc5e4eb5b/volumes" Nov 04 01:36:03.993745 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:03.993725 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c583de-adfc-4b67-8b77-eeddbc699916" path="/var/lib/kubelet/pods/f2c583de-adfc-4b67-8b77-eeddbc699916/volumes" Nov 04 01:36:05.518129 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:05.518103 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:36:05.679132 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:05.679001 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:36:05.818583 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:05.816611 4852 generic.go:356] "Generic (PLEG): container finished" podID="df08c3cb-cf91-4bec-a39d-c32e07739abf" containerID="2bc3fdbad492290277c5ab697666237aba16f9910ad6ae894f80497a5fa6615c" exitCode=0 Nov 04 01:36:05.818583 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:05.816700 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" event={"ID":"df08c3cb-cf91-4bec-a39d-c32e07739abf","Type":"ContainerDied","Data":"2bc3fdbad492290277c5ab697666237aba16f9910ad6ae894f80497a5fa6615c"} Nov 04 01:36:05.822590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:05.819010 4852 generic.go:356] "Generic (PLEG): container finished" podID="897931d4-4332-4739-8bd8-93433e6611be" containerID="b87cf4bd9ba0921b91d73becb244925c229605d58aa51a0f99f3cff4dc530e31" exitCode=0 Nov 04 01:36:05.822590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:05.819092 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" event={"ID":"897931d4-4332-4739-8bd8-93433e6611be","Type":"ContainerDied","Data":"b87cf4bd9ba0921b91d73becb244925c229605d58aa51a0f99f3cff4dc530e31"} Nov 04 01:36:05.822590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:05.821258 4852 generic.go:356] "Generic (PLEG): container finished" podID="1daad035-b4ae-4b3c-b522-cc267c4bf053" containerID="6419fe3a1faff4bcf7ecb634f2b37cacc7f3682c8663a1bdf53bfc7ed03d0c33" exitCode=0 Nov 04 01:36:05.822590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:05.821352 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" event={"ID":"1daad035-b4ae-4b3c-b522-cc267c4bf053","Type":"ContainerDied","Data":"6419fe3a1faff4bcf7ecb634f2b37cacc7f3682c8663a1bdf53bfc7ed03d0c33"} Nov 04 01:36:06.138588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.138061 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:36:06.148640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.143784 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/897931d4-4332-4739-8bd8-93433e6611be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/897931d4-4332-4739-8bd8-93433e6611be-istiod-ca-cert\") pod \"897931d4-4332-4739-8bd8-93433e6611be\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " Nov 04 01:36:06.148640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.143834 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/897931d4-4332-4739-8bd8-93433e6611be-kube-api-access-zwzgv\" (UniqueName: \"kubernetes.io/projected/897931d4-4332-4739-8bd8-93433e6611be-kube-api-access-zwzgv\") pod \"897931d4-4332-4739-8bd8-93433e6611be\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " Nov 04 01:36:06.148640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.143883 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-workload-certs\") pod \"897931d4-4332-4739-8bd8-93433e6611be\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " Nov 04 01:36:06.148640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.143904 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-credential-socket\") pod \"897931d4-4332-4739-8bd8-93433e6611be\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " Nov 04 01:36:06.148640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.143930 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/897931d4-4332-4739-8bd8-93433e6611be-istio-token\" (UniqueName: \"kubernetes.io/projected/897931d4-4332-4739-8bd8-93433e6611be-istio-token\") pod \"897931d4-4332-4739-8bd8-93433e6611be\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " Nov 04 01:36:06.148640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.143954 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-istio-envoy\") pod \"897931d4-4332-4739-8bd8-93433e6611be\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " Nov 04 01:36:06.148640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.143979 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-workload-socket\") pod \"897931d4-4332-4739-8bd8-93433e6611be\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " Nov 04 01:36:06.148640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.144018 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/897931d4-4332-4739-8bd8-93433e6611be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/897931d4-4332-4739-8bd8-93433e6611be-istio-podinfo\") pod \"897931d4-4332-4739-8bd8-93433e6611be\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " Nov 04 01:36:06.148640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.144039 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-istio-data\") pod \"897931d4-4332-4739-8bd8-93433e6611be\" (UID: \"897931d4-4332-4739-8bd8-93433e6611be\") " Nov 04 01:36:06.148640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.145656 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-istio-data" pod "897931d4-4332-4739-8bd8-93433e6611be" (UID: "897931d4-4332-4739-8bd8-93433e6611be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.148640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.147813 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/897931d4-4332-4739-8bd8-93433e6611be-istiod-ca-cert" pod "897931d4-4332-4739-8bd8-93433e6611be" (UID: "897931d4-4332-4739-8bd8-93433e6611be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:06.148640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.148057 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-workload-certs" pod "897931d4-4332-4739-8bd8-93433e6611be" (UID: "897931d4-4332-4739-8bd8-93433e6611be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.148640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.148222 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-credential-socket" pod "897931d4-4332-4739-8bd8-93433e6611be" (UID: "897931d4-4332-4739-8bd8-93433e6611be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.148640 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.148380 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-workload-socket" pod "897931d4-4332-4739-8bd8-93433e6611be" (UID: "897931d4-4332-4739-8bd8-93433e6611be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.153687 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.153660 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-istio-envoy" pod "897931d4-4332-4739-8bd8-93433e6611be" (UID: "897931d4-4332-4739-8bd8-93433e6611be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.153687 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.153663 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/897931d4-4332-4739-8bd8-93433e6611be-istio-podinfo" pod "897931d4-4332-4739-8bd8-93433e6611be" (UID: "897931d4-4332-4739-8bd8-93433e6611be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:36:06.153886 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.153737 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/897931d4-4332-4739-8bd8-93433e6611be-istio-token" pod "897931d4-4332-4739-8bd8-93433e6611be" (UID: "897931d4-4332-4739-8bd8-93433e6611be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:06.153886 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.153734 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/897931d4-4332-4739-8bd8-93433e6611be-kube-api-access-zwzgv" pod "897931d4-4332-4739-8bd8-93433e6611be" (UID: "897931d4-4332-4739-8bd8-93433e6611be"). InnerVolumeSpecName "kube-api-access-zwzgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:06.245111 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.245073 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/897931d4-4332-4739-8bd8-93433e6611be-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.245111 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.245107 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwzgv\" (UniqueName: \"kubernetes.io/projected/897931d4-4332-4739-8bd8-93433e6611be-kube-api-access-zwzgv\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.245349 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.245121 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.245349 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.245134 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.245349 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.245146 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/897931d4-4332-4739-8bd8-93433e6611be-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.245349 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.245156 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.245349 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.245168 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.245349 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.245179 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/897931d4-4332-4739-8bd8-93433e6611be-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.245349 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.245191 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/897931d4-4332-4739-8bd8-93433e6611be-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.400589 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.394733 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:36:06.439606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.437105 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:36:06.447485 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.447448 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-workload-socket\") pod \"1daad035-b4ae-4b3c-b522-cc267c4bf053\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " Nov 04 01:36:06.447675 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.447498 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-credential-socket\") pod \"df08c3cb-cf91-4bec-a39d-c32e07739abf\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " Nov 04 01:36:06.447675 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.447544 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-podinfo\") pod \"1daad035-b4ae-4b3c-b522-cc267c4bf053\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " Nov 04 01:36:06.447675 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.447595 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df08c3cb-cf91-4bec-a39d-c32e07739abf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df08c3cb-cf91-4bec-a39d-c32e07739abf-istiod-ca-cert\") pod \"df08c3cb-cf91-4bec-a39d-c32e07739abf\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " Nov 04 01:36:06.447675 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.447660 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1daad035-b4ae-4b3c-b522-cc267c4bf053-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1daad035-b4ae-4b3c-b522-cc267c4bf053-istiod-ca-cert\") pod \"1daad035-b4ae-4b3c-b522-cc267c4bf053\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " Nov 04 01:36:06.447927 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.447718 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-token\" (UniqueName: \"kubernetes.io/projected/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-token\") pod \"1daad035-b4ae-4b3c-b522-cc267c4bf053\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " Nov 04 01:36:06.447927 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.447756 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-data\") pod \"1daad035-b4ae-4b3c-b522-cc267c4bf053\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " Nov 04 01:36:06.447927 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.447789 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-envoy\") pod \"df08c3cb-cf91-4bec-a39d-c32e07739abf\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " Nov 04 01:36:06.447927 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.447824 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-data\") pod \"df08c3cb-cf91-4bec-a39d-c32e07739abf\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " Nov 04 01:36:06.447927 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.447850 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-credential-socket" pod "df08c3cb-cf91-4bec-a39d-c32e07739abf" (UID: "df08c3cb-cf91-4bec-a39d-c32e07739abf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.447927 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.447870 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-workload-certs\") pod \"df08c3cb-cf91-4bec-a39d-c32e07739abf\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " Nov 04 01:36:06.447927 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.447911 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1daad035-b4ae-4b3c-b522-cc267c4bf053-kube-api-access-26phm\" (UniqueName: \"kubernetes.io/projected/1daad035-b4ae-4b3c-b522-cc267c4bf053-kube-api-access-26phm\") pod \"1daad035-b4ae-4b3c-b522-cc267c4bf053\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " Nov 04 01:36:06.448375 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.447938 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-token\" (UniqueName: \"kubernetes.io/projected/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-token\") pod \"df08c3cb-cf91-4bec-a39d-c32e07739abf\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " Nov 04 01:36:06.448375 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.447965 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-credential-socket\") pod \"1daad035-b4ae-4b3c-b522-cc267c4bf053\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " Nov 04 01:36:06.448375 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.448000 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-workload-socket" pod "1daad035-b4ae-4b3c-b522-cc267c4bf053" (UID: "1daad035-b4ae-4b3c-b522-cc267c4bf053"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.448375 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.448008 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-workload-certs\") pod \"1daad035-b4ae-4b3c-b522-cc267c4bf053\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " Nov 04 01:36:06.448375 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.448055 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df08c3cb-cf91-4bec-a39d-c32e07739abf-kube-api-access-4bkhm\" (UniqueName: \"kubernetes.io/projected/df08c3cb-cf91-4bec-a39d-c32e07739abf-kube-api-access-4bkhm\") pod \"df08c3cb-cf91-4bec-a39d-c32e07739abf\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " Nov 04 01:36:06.448375 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.448097 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-workload-socket\") pod \"df08c3cb-cf91-4bec-a39d-c32e07739abf\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " Nov 04 01:36:06.448375 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.448124 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-podinfo\") pod \"df08c3cb-cf91-4bec-a39d-c32e07739abf\" (UID: \"df08c3cb-cf91-4bec-a39d-c32e07739abf\") " Nov 04 01:36:06.448375 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.448154 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-envoy\") pod \"1daad035-b4ae-4b3c-b522-cc267c4bf053\" (UID: \"1daad035-b4ae-4b3c-b522-cc267c4bf053\") " Nov 04 01:36:06.448886 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.448440 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.448886 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.448458 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.449772 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.449742 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df08c3cb-cf91-4bec-a39d-c32e07739abf-istiod-ca-cert" pod "df08c3cb-cf91-4bec-a39d-c32e07739abf" (UID: "df08c3cb-cf91-4bec-a39d-c32e07739abf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:06.449892 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.449832 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-workload-certs" pod "df08c3cb-cf91-4bec-a39d-c32e07739abf" (UID: "df08c3cb-cf91-4bec-a39d-c32e07739abf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.450692 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.449958 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-credential-socket" pod "1daad035-b4ae-4b3c-b522-cc267c4bf053" (UID: "1daad035-b4ae-4b3c-b522-cc267c4bf053"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.450692 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.449965 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-workload-socket" pod "df08c3cb-cf91-4bec-a39d-c32e07739abf" (UID: "df08c3cb-cf91-4bec-a39d-c32e07739abf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.450692 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.450448 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-data" pod "1daad035-b4ae-4b3c-b522-cc267c4bf053" (UID: "1daad035-b4ae-4b3c-b522-cc267c4bf053"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.451160 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.451041 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-data" pod "df08c3cb-cf91-4bec-a39d-c32e07739abf" (UID: "df08c3cb-cf91-4bec-a39d-c32e07739abf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.451160 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.451098 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-envoy" pod "df08c3cb-cf91-4bec-a39d-c32e07739abf" (UID: "df08c3cb-cf91-4bec-a39d-c32e07739abf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.451160 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.451142 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-workload-certs" pod "1daad035-b4ae-4b3c-b522-cc267c4bf053" (UID: "1daad035-b4ae-4b3c-b522-cc267c4bf053"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.451553 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.451528 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1daad035-b4ae-4b3c-b522-cc267c4bf053-istiod-ca-cert" pod "1daad035-b4ae-4b3c-b522-cc267c4bf053" (UID: "1daad035-b4ae-4b3c-b522-cc267c4bf053"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:06.452529 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.452488 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-podinfo" pod "1daad035-b4ae-4b3c-b522-cc267c4bf053" (UID: "1daad035-b4ae-4b3c-b522-cc267c4bf053"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:36:06.452791 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.452699 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-envoy" pod "1daad035-b4ae-4b3c-b522-cc267c4bf053" (UID: "1daad035-b4ae-4b3c-b522-cc267c4bf053"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.452791 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.452763 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df08c3cb-cf91-4bec-a39d-c32e07739abf-kube-api-access-4bkhm" pod "df08c3cb-cf91-4bec-a39d-c32e07739abf" (UID: "df08c3cb-cf91-4bec-a39d-c32e07739abf"). InnerVolumeSpecName "kube-api-access-4bkhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:06.453461 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.453431 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1daad035-b4ae-4b3c-b522-cc267c4bf053-kube-api-access-26phm" pod "1daad035-b4ae-4b3c-b522-cc267c4bf053" (UID: "1daad035-b4ae-4b3c-b522-cc267c4bf053"). InnerVolumeSpecName "kube-api-access-26phm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:06.453804 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.453780 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-token" pod "1daad035-b4ae-4b3c-b522-cc267c4bf053" (UID: "1daad035-b4ae-4b3c-b522-cc267c4bf053"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:06.454125 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.454104 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-token" pod "df08c3cb-cf91-4bec-a39d-c32e07739abf" (UID: "df08c3cb-cf91-4bec-a39d-c32e07739abf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:06.454485 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.454466 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-podinfo" pod "df08c3cb-cf91-4bec-a39d-c32e07739abf" (UID: "df08c3cb-cf91-4bec-a39d-c32e07739abf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:36:06.549816 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.549788 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.550126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.549822 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.550126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.549837 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.550126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.549851 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.550126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.549868 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.550126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.549886 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26phm\" (UniqueName: \"kubernetes.io/projected/1daad035-b4ae-4b3c-b522-cc267c4bf053-kube-api-access-26phm\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.550126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.549901 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.550126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.549914 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.550126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.549929 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.550126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.549943 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bkhm\" (UniqueName: \"kubernetes.io/projected/df08c3cb-cf91-4bec-a39d-c32e07739abf-kube-api-access-4bkhm\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.550126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.549957 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df08c3cb-cf91-4bec-a39d-c32e07739abf-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.550126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.549970 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df08c3cb-cf91-4bec-a39d-c32e07739abf-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.550126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.549985 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.550126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.549998 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1daad035-b4ae-4b3c-b522-cc267c4bf053-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.550126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.550012 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df08c3cb-cf91-4bec-a39d-c32e07739abf-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.550126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.550026 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1daad035-b4ae-4b3c-b522-cc267c4bf053-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.615633 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.612888 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:36:06.651178 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.651155 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-data\") pod \"fc3454a1-295f-441f-9e75-54d0dc18bfea\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " Nov 04 01:36:06.651338 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.651196 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc3454a1-295f-441f-9e75-54d0dc18bfea-kube-api-access-wj4fp\" (UniqueName: \"kubernetes.io/projected/fc3454a1-295f-441f-9e75-54d0dc18bfea-kube-api-access-wj4fp\") pod \"fc3454a1-295f-441f-9e75-54d0dc18bfea\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " Nov 04 01:36:06.651338 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.651223 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc3454a1-295f-441f-9e75-54d0dc18bfea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc3454a1-295f-441f-9e75-54d0dc18bfea-istiod-ca-cert\") pod \"fc3454a1-295f-441f-9e75-54d0dc18bfea\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " Nov 04 01:36:06.651338 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.651241 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-credential-socket\") pod \"fc3454a1-295f-441f-9e75-54d0dc18bfea\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " Nov 04 01:36:06.651338 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.651259 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-token\" (UniqueName: \"kubernetes.io/projected/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-token\") pod \"fc3454a1-295f-441f-9e75-54d0dc18bfea\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " Nov 04 01:36:06.651503 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.651426 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-data" pod "fc3454a1-295f-441f-9e75-54d0dc18bfea" (UID: "fc3454a1-295f-441f-9e75-54d0dc18bfea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.651503 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.651483 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-podinfo\") pod \"fc3454a1-295f-441f-9e75-54d0dc18bfea\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " Nov 04 01:36:06.651625 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.651514 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-workload-socket\") pod \"fc3454a1-295f-441f-9e75-54d0dc18bfea\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " Nov 04 01:36:06.651625 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.651535 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-workload-certs\") pod \"fc3454a1-295f-441f-9e75-54d0dc18bfea\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " Nov 04 01:36:06.651625 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.651616 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-envoy\") pod \"fc3454a1-295f-441f-9e75-54d0dc18bfea\" (UID: \"fc3454a1-295f-441f-9e75-54d0dc18bfea\") " Nov 04 01:36:06.651796 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.651746 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-workload-socket" pod "fc3454a1-295f-441f-9e75-54d0dc18bfea" (UID: "fc3454a1-295f-441f-9e75-54d0dc18bfea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.651933 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.651907 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-credential-socket" pod "fc3454a1-295f-441f-9e75-54d0dc18bfea" (UID: "fc3454a1-295f-441f-9e75-54d0dc18bfea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.651980 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.651934 4852 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-data\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.651980 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.651911 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc3454a1-295f-441f-9e75-54d0dc18bfea-istiod-ca-cert" pod "fc3454a1-295f-441f-9e75-54d0dc18bfea" (UID: "fc3454a1-295f-441f-9e75-54d0dc18bfea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:06.651980 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.651947 4852 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-workload-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.652238 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.652180 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-workload-certs" pod "fc3454a1-295f-441f-9e75-54d0dc18bfea" (UID: "fc3454a1-295f-441f-9e75-54d0dc18bfea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.654036 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.654009 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-envoy" pod "fc3454a1-295f-441f-9e75-54d0dc18bfea" (UID: "fc3454a1-295f-441f-9e75-54d0dc18bfea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.654036 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.654018 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3454a1-295f-441f-9e75-54d0dc18bfea-kube-api-access-wj4fp" pod "fc3454a1-295f-441f-9e75-54d0dc18bfea" (UID: "fc3454a1-295f-441f-9e75-54d0dc18bfea"). InnerVolumeSpecName "kube-api-access-wj4fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:06.654193 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.654170 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-podinfo" pod "fc3454a1-295f-441f-9e75-54d0dc18bfea" (UID: "fc3454a1-295f-441f-9e75-54d0dc18bfea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:36:06.654290 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.654272 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-token" pod "fc3454a1-295f-441f-9e75-54d0dc18bfea" (UID: "fc3454a1-295f-441f-9e75-54d0dc18bfea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:06.752584 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.752537 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj4fp\" (UniqueName: \"kubernetes.io/projected/fc3454a1-295f-441f-9e75-54d0dc18bfea-kube-api-access-wj4fp\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.752584 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.752578 4852 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc3454a1-295f-441f-9e75-54d0dc18bfea-istiod-ca-cert\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.752584 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.752588 4852 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-credential-socket\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.752799 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.752597 4852 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-token\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.752799 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.752605 4852 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-podinfo\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.752799 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.752613 4852 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-workload-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.752799 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.752621 4852 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc3454a1-295f-441f-9e75-54d0dc18bfea-istio-envoy\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.841981 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.841938 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" event={"ID":"df08c3cb-cf91-4bec-a39d-c32e07739abf","Type":"ContainerDied","Data":"3b33be0c98fc29c4a216f6fefccb1cfa621ff9b07672b114c46d7e063ef8a05f"} Nov 04 01:36:06.842172 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.842030 4852 scope.go:122] "RemoveContainer" containerID="2bc3fdbad492290277c5ab697666237aba16f9910ad6ae894f80497a5fa6615c" Nov 04 01:36:06.842172 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.841962 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f" Nov 04 01:36:06.843926 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.843663 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" event={"ID":"897931d4-4332-4739-8bd8-93433e6611be","Type":"ContainerDied","Data":"14189b7354b84dde9c5ee51fb5d482033c493d196ba7130b2701de4f26152c2b"} Nov 04 01:36:06.843926 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.843764 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw" Nov 04 01:36:06.845361 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.845287 4852 generic.go:356] "Generic (PLEG): container finished" podID="fc3454a1-295f-441f-9e75-54d0dc18bfea" containerID="84dffa77ab7ca1db3a1a9561f28a83d77af02e281ee4cac3463e443e6a417331" exitCode=0 Nov 04 01:36:06.845361 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.845308 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" Nov 04 01:36:06.845361 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.845344 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" event={"ID":"fc3454a1-295f-441f-9e75-54d0dc18bfea","Type":"ContainerDied","Data":"84dffa77ab7ca1db3a1a9561f28a83d77af02e281ee4cac3463e443e6a417331"} Nov 04 01:36:06.845692 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.845391 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7" event={"ID":"fc3454a1-295f-441f-9e75-54d0dc18bfea","Type":"ContainerDied","Data":"c39973b11b77639dbce887555b38939a3cf7ff05e325ad426669a53a98c096a5"} Nov 04 01:36:06.847924 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.847905 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" Nov 04 01:36:06.847924 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.847915 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt" event={"ID":"1daad035-b4ae-4b3c-b522-cc267c4bf053","Type":"ContainerDied","Data":"ccd90436ba78c02ac1b4194da83529b098fe741e2de4862e90383ecb46f3c66a"} Nov 04 01:36:06.853156 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.852939 4852 scope.go:122] "RemoveContainer" containerID="b87cf4bd9ba0921b91d73becb244925c229605d58aa51a0f99f3cff4dc530e31" Nov 04 01:36:06.919653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.905273 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw"] Nov 04 01:36:06.919653 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.916625 4852 scope.go:122] "RemoveContainer" containerID="84dffa77ab7ca1db3a1a9561f28a83d77af02e281ee4cac3463e443e6a417331" Nov 04 01:36:06.944492 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.944454 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-56dnw"] Nov 04 01:36:06.947344 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.947326 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7"] Nov 04 01:36:06.949242 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.949216 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-5zpv7"] Nov 04 01:36:06.951223 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.951193 4852 scope.go:122] "RemoveContainer" containerID="84dffa77ab7ca1db3a1a9561f28a83d77af02e281ee4cac3463e443e6a417331" Nov 04 01:36:06.951483 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:36:06.951462 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84dffa77ab7ca1db3a1a9561f28a83d77af02e281ee4cac3463e443e6a417331\": container with ID starting with 84dffa77ab7ca1db3a1a9561f28a83d77af02e281ee4cac3463e443e6a417331 not found: ID does not exist" containerID="84dffa77ab7ca1db3a1a9561f28a83d77af02e281ee4cac3463e443e6a417331" Nov 04 01:36:06.951535 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.951504 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84dffa77ab7ca1db3a1a9561f28a83d77af02e281ee4cac3463e443e6a417331"} err="failed to get container status \"84dffa77ab7ca1db3a1a9561f28a83d77af02e281ee4cac3463e443e6a417331\": rpc error: code = NotFound desc = could not find container \"84dffa77ab7ca1db3a1a9561f28a83d77af02e281ee4cac3463e443e6a417331\": container with ID starting with 84dffa77ab7ca1db3a1a9561f28a83d77af02e281ee4cac3463e443e6a417331 not found: ID does not exist" Nov 04 01:36:06.951535 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.951523 4852 scope.go:122] "RemoveContainer" containerID="6419fe3a1faff4bcf7ecb634f2b37cacc7f3682c8663a1bdf53bfc7ed03d0c33" Nov 04 01:36:06.960748 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.957883 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f"] Nov 04 01:36:06.969585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.967667 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l7h5f"] Nov 04 01:36:06.988621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.981625 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt"] Nov 04 01:36:06.989209 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.988803 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2ttwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:36:06.989209 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:06.988853 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2ttwk" podUID="266b7c8e-ffa2-410b-a592-a96f3e034103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:36:07.012584 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:07.006645 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-ffmnt"] Nov 04 01:36:07.992384 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:07.992350 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1daad035-b4ae-4b3c-b522-cc267c4bf053" path="/var/lib/kubelet/pods/1daad035-b4ae-4b3c-b522-cc267c4bf053/volumes" Nov 04 01:36:07.992782 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:07.992769 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="897931d4-4332-4739-8bd8-93433e6611be" path="/var/lib/kubelet/pods/897931d4-4332-4739-8bd8-93433e6611be/volumes" Nov 04 01:36:07.993137 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:07.993124 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df08c3cb-cf91-4bec-a39d-c32e07739abf" path="/var/lib/kubelet/pods/df08c3cb-cf91-4bec-a39d-c32e07739abf/volumes" Nov 04 01:36:07.993792 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:07.993777 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc3454a1-295f-441f-9e75-54d0dc18bfea" path="/var/lib/kubelet/pods/fc3454a1-295f-441f-9e75-54d0dc18bfea/volumes" Nov 04 01:36:20.658182 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:20.658147 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:36:20.809148 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:20.809114 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:36:29.865497 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:36:29.865458 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:36:37.123665 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:37.123626 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:36:38.282544 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:38.282509 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:36:38.969483 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:38.969449 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:36:51.933386 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:51.933312 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:36:55.929695 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:55.929660 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:36:56.070457 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:36:56.070424 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:36:59.865963 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:36:59.865926 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:37:13.921119 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:37:13.921087 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:37:29.865257 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:37:29.865222 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:37:29.952789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:37:29.952762 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:37:33.837009 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:37:33.836955 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:37:34.169648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:37:34.162667 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-tqg84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:37:34.169648 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:37:34.162734 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" podUID="c13769b4-f82d-460e-aa0f-0e882b40346c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:37:37.334993 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:37:37.334962 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:37:38.347043 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:37:38.347009 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:37:39.267786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:37:39.267738 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:37:39.413133 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:37:39.413091 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:37:47.910053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:37:47.910019 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:37:52.321350 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:37:52.321315 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:37:58.025651 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:37:58.025619 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:37:58.172039 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:37:58.172012 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:37:59.865665 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:37:59.865624 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:37:59.921322 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:37:59.921290 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:38:07.337938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:38:07.337881 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:38:07.499229 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:38:07.499201 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:38:22.528039 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:38:22.527955 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:38:29.864366 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:38:29.864332 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:38:32.609990 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:38:32.609955 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:38:37.672590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:38:37.671713 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lkkwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:38:37.672590 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:38:37.671799 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lkkwt" podUID="a232696a-6a25-4803-a2a0-a9888279273c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:38:38.316421 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:38:38.316387 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:38:38.419140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:38:38.419113 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 01:38:38.425820 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:38:38.425799 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 01:38:38.523371 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:38:38.523338 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:38:53.546978 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:38:53.546939 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:38:55.990806 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:38:55.990728 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:38:57.581867 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:38:57.581827 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:38:59.866765 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:38:59.866722 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:39:01.315457 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:39:01.315423 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:39:12.810243 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:39:12.810206 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:39:26.464176 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:39:26.464141 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:39:29.922588 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:39:29.920811 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:39:30.490397 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:39:30.490340 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:39:34.035911 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:39:34.035872 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:39:35.388158 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:39:35.388101 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:39:35.700607 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:39:35.700484 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:39:38.541527 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:39:38.541492 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:39:51.604036 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:39:51.603957 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:39:56.854647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:39:56.854609 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:39:57.441446 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:39:57.441415 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:39:57.643285 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:39:57.643250 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:39:59.865365 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:39:59.865318 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:40:01.665943 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:40:01.665901 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:40:01.813088 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:40:01.813036 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:40:10.014979 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:40:10.014945 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:40:13.308351 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:40:13.308304 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:40:17.446352 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:40:17.446303 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:40:24.021250 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:40:24.021212 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw"] Nov 04 01:40:24.024177 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:40:24.024152 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-616cb1ce-k7mbw"] Nov 04 01:40:25.992448 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:40:25.992412 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46120899-8e38-4afd-8e30-787092a30fc8" path="/var/lib/kubelet/pods/46120899-8e38-4afd-8e30-787092a30fc8/volumes" Nov 04 01:40:27.184254 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:40:27.184221 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:40:29.864917 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:40:29.864878 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:40:33.114935 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:40:33.114895 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:40:38.596609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:40:38.596549 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:40:43.344225 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:40:43.344195 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:40:58.976792 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:40:58.976758 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:40:59.925772 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:40:59.925731 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:41:03.177338 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:03.177309 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:03.333157 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:03.333123 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:41:03.659483 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:03.659440 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:03.815971 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:03.815907 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:07.283764 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:07.283707 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:07.432203 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:07.432174 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:08.444142 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:08.444099 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:08.610022 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:08.609987 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:09.243191 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:09.243134 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:10.050119 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:10.050057 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:10.727689 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:10.727640 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:10.844542 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:10.844509 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:41:11.531861 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:11.531829 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:12.197778 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:12.197735 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:12.928388 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:12.928347 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:13.332086 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:13.332060 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:13.596703 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:13.596607 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:14.225200 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:14.225167 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:14.829462 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:14.822807 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:15.440614 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:15.440525 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:16.058629 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:16.058591 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:16.728155 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:16.728105 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:17.466096 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:17.466065 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:18.330934 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:18.330881 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:19.129469 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:19.129436 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:19.835811 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:19.835761 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:20.511825 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:20.511791 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:21.129337 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:21.129307 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:21.823260 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:21.823207 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:22.169967 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:22.169890 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:22.514628 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:22.514550 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:22.779826 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:22.779746 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:22.941598 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:22.938742 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:23.095750 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:23.095676 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:23.363455 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:23.363378 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:24.147660 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:24.147633 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:24.483401 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:24.483319 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:24.932587 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:24.932286 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:25.843367 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:25.843329 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:26.334740 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:26.334703 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:41:26.662051 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:26.661972 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:27.418108 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:27.418076 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:28.030842 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:28.030806 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:28.812782 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:28.812751 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:29.422833 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:29.422800 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:29.865198 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:41:29.865166 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:41:30.103602 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:30.102736 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:30.756403 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:30.756370 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:30.892666 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:30.892628 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:31.506430 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:31.506344 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:32.128962 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:32.128917 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:32.812887 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:32.812853 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:33.518535 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:33.518496 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:34.228585 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:34.226645 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:35.979248 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:35.979212 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:36.114947 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:36.114917 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:36.246419 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:36.246340 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:36.835302 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:36.835265 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:37.510624 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:37.509080 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:38.183117 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:38.183048 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:38.661637 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:38.661607 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:38.817991 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:38.817905 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:39.532322 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:39.532250 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:39.560556 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:39.560527 4852 scope.go:122] "RemoveContainer" containerID="a995b495234bda418206abb2d98a21c85c94ea2d6a9e1503fd1b14d0cf618659" Nov 04 01:41:40.207712 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:40.207677 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:41.005666 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:41.005633 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:41.528344 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:41.528314 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:41:41.897378 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:41.897289 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:42.710333 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:42.710293 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:43.422836 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:43.422782 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:44.218545 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:44.218510 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:44.826852 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:44.826803 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:45.563365 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:45.563329 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:46.225446 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:46.225415 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:46.938647 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:46.937707 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:47.717658 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:47.717626 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:48.442747 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:48.442715 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:49.125944 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:49.125910 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:49.827980 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:49.827947 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:50.513836 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:50.513788 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:51.210060 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:51.210027 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:51.918109 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:51.918073 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:52.549489 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:52.549458 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:53.210206 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:53.210171 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:53.827548 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:53.826533 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:54.521222 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:54.521166 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:55.304690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:55.301271 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:56.025156 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:56.024789 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:56.642460 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:56.642426 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:57.314265 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:57.314230 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:58.105365 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:58.105323 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:58.818305 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:58.818262 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:59.519032 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:41:59.518995 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:41:59.865473 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:41:59.865396 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:42:00.133315 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:00.133207 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:00.818872 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:00.818829 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:01.475855 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:01.475820 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:02.120837 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:02.120803 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:02.731038 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:02.730997 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:03.408717 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:03.408686 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:04.032876 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:04.032825 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:04.740731 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:04.740695 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:04.884119 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:04.884087 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:05.279786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:05.279750 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:05.453119 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:05.453069 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:42:05.518403 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:05.518372 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:05.661193 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:05.661119 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:06.138007 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:06.137972 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:06.931072 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:06.931031 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:07.628629 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:07.628597 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:08.314035 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:08.313936 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:09.115038 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:09.114928 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:09.745999 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:09.745955 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:09.883095 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:09.883060 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:10.423356 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:10.423317 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:10.863759 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:10.863728 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:11.021164 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:11.021135 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:11.256843 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:11.256816 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:11.431760 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:11.431709 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:11.995948 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:11.995898 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:13.553937 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:13.553905 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:42:16.888133 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:16.888099 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:17.076480 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:17.076452 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:25.034902 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:25.034861 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:29.864557 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:42:29.864520 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:42:38.701631 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:38.701598 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:42.190353 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:42.190262 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:42.335145 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:42.335113 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:43.557123 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:43.557085 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:43.735053 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:43.735020 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:44.440695 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:44.440662 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:45.133959 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:45.133923 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:45.820555 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:45.820496 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:46.321050 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:46.321018 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:42:46.526989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:46.526953 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:46.775299 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:46.775262 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:47.237416 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:47.237335 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:47.751461 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:47.751408 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:48.036686 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:48.036625 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:48.847770 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:48.847727 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:49.637881 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:49.637840 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:50.324151 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:50.324115 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:50.984387 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:50.984339 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:51.150293 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:51.150229 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:42:51.648454 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:51.648408 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:52.333030 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:52.332997 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:53.148512 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:53.148472 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:53.782097 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:53.782066 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:53.930966 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:53.930930 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:54.516500 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:54.516457 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:55.222426 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:55.222393 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:55.916843 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:55.916807 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:56.610368 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:56.610336 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:57.243593 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:57.243537 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:57.922636 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:57.922264 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:58.581673 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:58.581162 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:59.305927 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:59.305888 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:42:59.865409 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:42:59.865372 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:42:59.921438 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:42:59.921405 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:43:00.612095 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:43:00.612038 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:43:01.333448 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:43:01.333412 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:43:02.022773 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:43:02.022732 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:43:02.680349 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:43:02.680315 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:43:03.051094 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:43:03.051066 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:43:03.202225 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:43:03.202193 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:43:03.353619 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:43:03.353514 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:43:03.489446 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:43:03.489410 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:43:03.857017 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:43:03.856982 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:43:05.123751 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:43:05.123713 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:43:16.516775 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:43:16.516741 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:43:16.601110 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:43:16.601082 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:43:16.750894 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:43:16.750863 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:43:29.866097 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:43:29.866059 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:43:38.446783 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:43:38.446756 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 01:43:38.450977 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:43:38.450957 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 01:43:38.734149 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:43:38.734082 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:43:59.865678 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:43:59.865640 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:44:00.491297 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:44:00.491266 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:44:03.167218 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:44:03.167188 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:44:19.369244 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:44:19.369163 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:44:29.865687 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:44:29.865650 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:44:38.787878 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:44:38.787833 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:44:39.621069 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:44:39.621030 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:44:55.127863 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:44:55.127828 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:44:59.930589 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:44:59.926824 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:45:00.053702 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:00.053664 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:45:03.255422 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:03.255388 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:45:13.074245 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:13.074206 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:13.494336 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:13.494252 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:13.633457 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:13.633429 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:16.860433 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:16.860406 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:17.032201 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:17.032168 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:18.079154 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:18.079126 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:18.298265 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:18.298231 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:18.973344 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:18.973309 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:19.932175 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:19.932146 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:20.734250 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:20.734222 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:20.772442 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:20.772417 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:45:21.614362 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:21.614325 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:22.319801 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:22.319753 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:23.028279 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:23.028245 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:23.623751 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:23.623714 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:24.501998 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:24.501966 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:25.124228 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:25.121007 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:25.921487 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:25.921447 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:26.482494 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:26.482420 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:27.227671 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:27.227634 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:27.815525 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:27.815485 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:28.482167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:28.482080 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:29.250230 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:29.250195 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:29.866205 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:45:29.866166 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:45:29.947804 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:29.947763 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:30.729988 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:30.729958 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:31.345074 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:31.345046 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:31.813472 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:31.813443 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:32.040119 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:32.039472 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:32.396045 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:32.396015 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:32.548994 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:32.548965 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:32.721756 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:32.721671 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:32.877729 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:32.877699 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:33.542845 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:33.542787 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:34.212405 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:34.212373 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:34.372581 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:34.372531 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:35.031403 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:35.031366 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:35.732479 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:35.732432 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:36.068167 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:36.068137 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:45:36.593308 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:36.593275 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:37.332091 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:37.332050 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:38.017609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:38.017530 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:38.839339 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:38.839306 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:38.999453 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:38.999411 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:39.534962 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:39.534933 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:40.343794 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:40.343758 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:41.094168 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:41.094137 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:41.958790 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:41.958697 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:42.745220 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:42.745187 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:43.443903 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:43.443864 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:44.213721 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:44.213676 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:44.925039 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:44.924928 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:45.705935 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:45.705898 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:46.441519 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:46.441476 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:47.145140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:47.145101 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:47.844011 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:47.843960 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:48.528535 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:48.528503 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:49.224603 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:49.224551 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:50.045958 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:50.045922 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:50.823068 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:50.823034 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:51.534292 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:51.534260 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:52.041607 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:52.041551 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:45:52.249742 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:52.249709 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:53.077979 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:53.077944 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:53.819683 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:53.819649 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:54.622706 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:54.622668 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:55.424518 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:55.424480 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:56.026305 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:56.026041 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:56.740219 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:56.740188 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:57.351070 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:57.351035 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:58.031900 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:58.031857 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:58.736342 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:58.736301 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:59.631128 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:45:59.631094 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:45:59.866102 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:45:59.866065 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:46:00.337942 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:00.337903 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:01.174681 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:01.174649 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:01.824781 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:01.824747 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:02.523095 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:02.523066 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:03.232025 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:03.231992 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:03.925904 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:03.925866 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:04.739492 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:04.739460 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:05.440494 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:05.440465 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:06.114145 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:06.114109 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:06.733107 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:06.733066 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:07.438426 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:07.438151 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:08.073822 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:08.073361 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:08.729067 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:08.729024 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:09.323626 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:09.323589 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:10.016994 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:10.016958 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:10.651068 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:10.651036 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:11.437203 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:11.437169 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:12.267306 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:12.267271 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:12.942994 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:12.942960 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:13.724570 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:13.724523 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:14.112170 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:14.112132 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:14.257604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:14.257557 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:14.400123 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:14.400038 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:14.747097 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:14.747021 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:46:14.764507 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:14.764478 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:14.939940 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:14.939910 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:15.276207 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:15.276174 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:15.625645 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:15.623656 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:16.280885 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:16.280849 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:16.921773 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:16.921740 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:17.704179 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:17.704062 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:18.327622 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:18.327580 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:19.020951 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:19.020914 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:19.641724 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:19.641685 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:19.857616 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:19.857536 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:20.417366 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:20.417327 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:21.108382 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:21.108349 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:21.263575 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:21.263541 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:21.472107 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:21.472022 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:21.702664 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:21.702625 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:23.248168 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:23.248139 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:46:27.031092 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:27.031048 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:27.181621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:27.181588 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:28.918012 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:28.917979 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:29.865253 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:46:29.865218 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:46:38.879424 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:38.879392 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:52.028414 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:52.028380 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:52.194056 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:52.194024 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:53.222427 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:53.222393 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:53.427185 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:53.427149 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:54.029330 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:54.029080 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:54.725197 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:54.725158 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:55.426524 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:55.426485 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:55.630271 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:55.630221 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:46:56.277359 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:56.277318 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:56.966253 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:56.966217 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:57.473273 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:57.473230 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:57.939179 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:57.939145 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:58.428656 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:58.428628 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:58.658373 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:58.658345 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:46:58.851917 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:58.851886 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:59.552106 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:46:59.552070 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:46:59.865775 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:46:59.865682 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:47:00.263736 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:00.263701 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:00.934122 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:00.934083 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:01.691864 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:01.691833 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:02.421984 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:02.421950 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:03.255449 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:03.255417 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:03.947632 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:03.947586 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:04.704973 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:04.704939 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:04.882069 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:04.882041 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:05.444917 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:05.444890 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:06.118950 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:06.118858 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:06.951619 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:06.951581 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:07.813863 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:07.813828 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:08.351844 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:08.351815 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:09.028686 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:09.028648 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:09.991891 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:09.991853 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:10.472085 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:10.472052 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:11.237142 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:11.237085 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:11.680184 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:11.680111 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:11.816789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:11.816757 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:11.966835 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:11.966751 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:12.497297 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:12.497260 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:12.664735 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:12.664698 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:12.825805 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:12.825772 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:13.105389 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:13.105306 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:14.125200 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:14.125168 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:47:19.606011 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:19.605976 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:21.635786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:21.635753 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:47:29.866218 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:47:29.866176 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:47:33.788161 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:33.788114 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:35.342736 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:35.342695 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:47:39.081786 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:39.081752 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:47:51.246996 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:47:51.246959 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:47:59.866673 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:47:59.866640 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:48:01.697210 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:01.697168 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:02.661228 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:02.661188 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:03.054253 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:03.054217 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:48:09.372604 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:09.371885 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:48:14.939755 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:14.939720 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:48:19.965591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:19.959229 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:48:29.868215 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:48:29.868175 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:48:33.311077 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:33.311043 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:48:38.474868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:38.474840 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 01:48:38.475285 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:38.475124 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 01:48:39.139466 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:39.139425 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:39.277808 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:39.277782 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:41.791872 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:41.791790 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:42.292432 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.292254 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:42.538081 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.538054 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:42.990242 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990211 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-845d896dc4-nvcf6"] Nov 04 01:48:42.990678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990519 4852 cpu_manager.go:414] "containerMap: removing container" podUID="fc3454a1-295f-441f-9e75-54d0dc18bfea" containerName="istio-proxy" Nov 04 01:48:42.990678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990530 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc3454a1-295f-441f-9e75-54d0dc18bfea" containerName="istio-proxy" Nov 04 01:48:42.990678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990544 4852 cpu_manager.go:414] "containerMap: removing container" podUID="f2c583de-adfc-4b67-8b77-eeddbc699916" containerName="istio-proxy" Nov 04 01:48:42.990678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990548 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c583de-adfc-4b67-8b77-eeddbc699916" containerName="istio-proxy" Nov 04 01:48:42.990678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990577 4852 cpu_manager.go:414] "containerMap: removing container" podUID="cd9afd3b-9e83-4731-9b32-305bc5e4eb5b" containerName="istio-proxy" Nov 04 01:48:42.990678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990582 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd9afd3b-9e83-4731-9b32-305bc5e4eb5b" containerName="istio-proxy" Nov 04 01:48:42.990678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990593 4852 cpu_manager.go:414] "containerMap: removing container" podUID="897931d4-4332-4739-8bd8-93433e6611be" containerName="istio-proxy" Nov 04 01:48:42.990678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990598 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="897931d4-4332-4739-8bd8-93433e6611be" containerName="istio-proxy" Nov 04 01:48:42.990678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990608 4852 cpu_manager.go:414] "containerMap: removing container" podUID="1daad035-b4ae-4b3c-b522-cc267c4bf053" containerName="istio-proxy" Nov 04 01:48:42.990678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990612 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1daad035-b4ae-4b3c-b522-cc267c4bf053" containerName="istio-proxy" Nov 04 01:48:42.990678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990624 4852 cpu_manager.go:414] "containerMap: removing container" podUID="df08c3cb-cf91-4bec-a39d-c32e07739abf" containerName="istio-proxy" Nov 04 01:48:42.990678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990628 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df08c3cb-cf91-4bec-a39d-c32e07739abf" containerName="istio-proxy" Nov 04 01:48:42.990678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990639 4852 cpu_manager.go:414] "containerMap: removing container" podUID="6d1708da-d20c-474b-81d6-7eac1175396d" containerName="istio-proxy" Nov 04 01:48:42.990678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990644 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d1708da-d20c-474b-81d6-7eac1175396d" containerName="istio-proxy" Nov 04 01:48:42.990678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990650 4852 cpu_manager.go:414] "containerMap: removing container" podUID="c0958a62-b79b-4d8f-9a10-81acf1522245" containerName="istio-proxy" Nov 04 01:48:42.990678 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990655 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0958a62-b79b-4d8f-9a10-81acf1522245" containerName="istio-proxy" Nov 04 01:48:42.991187 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990703 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="897931d4-4332-4739-8bd8-93433e6611be" containerName="istio-proxy" Nov 04 01:48:42.991187 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990711 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="1daad035-b4ae-4b3c-b522-cc267c4bf053" containerName="istio-proxy" Nov 04 01:48:42.991187 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990718 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="df08c3cb-cf91-4bec-a39d-c32e07739abf" containerName="istio-proxy" Nov 04 01:48:42.991187 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990724 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d1708da-d20c-474b-81d6-7eac1175396d" containerName="istio-proxy" Nov 04 01:48:42.991187 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990731 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0958a62-b79b-4d8f-9a10-81acf1522245" containerName="istio-proxy" Nov 04 01:48:42.991187 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990737 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc3454a1-295f-441f-9e75-54d0dc18bfea" containerName="istio-proxy" Nov 04 01:48:42.991187 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990743 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c583de-adfc-4b67-8b77-eeddbc699916" containerName="istio-proxy" Nov 04 01:48:42.991187 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.990748 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd9afd3b-9e83-4731-9b32-305bc5e4eb5b" containerName="istio-proxy" Nov 04 01:48:42.993660 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:42.993641 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-845d896dc4-nvcf6" Nov 04 01:48:43.029155 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:43.029129 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-845d896dc4-nvcf6"] Nov 04 01:48:43.078508 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:43.078474 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f770fbd-3157-4e1d-bb7e-d9aadc559239-maas-api-tls\") pod \"maas-api-845d896dc4-nvcf6\" (UID: \"4f770fbd-3157-4e1d-bb7e-d9aadc559239\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-nvcf6" Nov 04 01:48:43.078690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:43.078519 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkmpg\" (UniqueName: \"kubernetes.io/projected/4f770fbd-3157-4e1d-bb7e-d9aadc559239-kube-api-access-jkmpg\") pod \"maas-api-845d896dc4-nvcf6\" (UID: \"4f770fbd-3157-4e1d-bb7e-d9aadc559239\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-nvcf6" Nov 04 01:48:43.078690 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:43.078553 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f770fbd-3157-4e1d-bb7e-d9aadc559239-openshift-service-ca\") pod \"maas-api-845d896dc4-nvcf6\" (UID: \"4f770fbd-3157-4e1d-bb7e-d9aadc559239\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-nvcf6" Nov 04 01:48:43.078769 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:43.078693 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f770fbd-3157-4e1d-bb7e-d9aadc559239-metrics-certs\") pod \"maas-api-845d896dc4-nvcf6\" (UID: \"4f770fbd-3157-4e1d-bb7e-d9aadc559239\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-nvcf6" Nov 04 01:48:43.179349 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:43.179316 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f770fbd-3157-4e1d-bb7e-d9aadc559239-metrics-certs\") pod \"maas-api-845d896dc4-nvcf6\" (UID: \"4f770fbd-3157-4e1d-bb7e-d9aadc559239\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-nvcf6" Nov 04 01:48:43.179511 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:43.179365 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f770fbd-3157-4e1d-bb7e-d9aadc559239-maas-api-tls\") pod \"maas-api-845d896dc4-nvcf6\" (UID: \"4f770fbd-3157-4e1d-bb7e-d9aadc559239\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-nvcf6" Nov 04 01:48:43.179511 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:43.179385 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkmpg\" (UniqueName: \"kubernetes.io/projected/4f770fbd-3157-4e1d-bb7e-d9aadc559239-kube-api-access-jkmpg\") pod \"maas-api-845d896dc4-nvcf6\" (UID: \"4f770fbd-3157-4e1d-bb7e-d9aadc559239\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-nvcf6" Nov 04 01:48:43.179511 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:43.179404 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f770fbd-3157-4e1d-bb7e-d9aadc559239-openshift-service-ca\") pod \"maas-api-845d896dc4-nvcf6\" (UID: \"4f770fbd-3157-4e1d-bb7e-d9aadc559239\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-nvcf6" Nov 04 01:48:43.180101 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:43.180068 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f770fbd-3157-4e1d-bb7e-d9aadc559239-openshift-service-ca\") pod \"maas-api-845d896dc4-nvcf6\" (UID: \"4f770fbd-3157-4e1d-bb7e-d9aadc559239\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-nvcf6" Nov 04 01:48:43.182013 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:43.181992 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f770fbd-3157-4e1d-bb7e-d9aadc559239-metrics-certs\") pod \"maas-api-845d896dc4-nvcf6\" (UID: \"4f770fbd-3157-4e1d-bb7e-d9aadc559239\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-nvcf6" Nov 04 01:48:43.182116 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:43.182028 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f770fbd-3157-4e1d-bb7e-d9aadc559239-maas-api-tls\") pod \"maas-api-845d896dc4-nvcf6\" (UID: \"4f770fbd-3157-4e1d-bb7e-d9aadc559239\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-nvcf6" Nov 04 01:48:43.190962 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:43.190936 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkmpg\" (UniqueName: \"kubernetes.io/projected/4f770fbd-3157-4e1d-bb7e-d9aadc559239-kube-api-access-jkmpg\") pod \"maas-api-845d896dc4-nvcf6\" (UID: \"4f770fbd-3157-4e1d-bb7e-d9aadc559239\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-nvcf6" Nov 04 01:48:43.305269 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:43.305242 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-845d896dc4-nvcf6" Nov 04 01:48:43.592016 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:43.591936 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-845d896dc4-nvcf6"] Nov 04 01:48:43.595069 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:48:43.595043 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f770fbd_3157_4e1d_bb7e_d9aadc559239.slice/crio-408ed0e35c5250a3ea648640596613cfe0a6b20d6985d8110827e69f90944219 WatchSource:0}: Error finding container 408ed0e35c5250a3ea648640596613cfe0a6b20d6985d8110827e69f90944219: Status 404 returned error can't find the container with id 408ed0e35c5250a3ea648640596613cfe0a6b20d6985d8110827e69f90944219 Nov 04 01:48:43.596381 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:43.596364 4852 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:48:44.395761 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:44.395724 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-845d896dc4-nvcf6" event={"ID":"4f770fbd-3157-4e1d-bb7e-d9aadc559239","Type":"ContainerStarted","Data":"be7934992725d740a0da9c6b1bcf7b6bd58b39c31bfe48d6dfdab472ff09fc1d"} Nov 04 01:48:44.396238 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:44.395783 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-845d896dc4-nvcf6" event={"ID":"4f770fbd-3157-4e1d-bb7e-d9aadc559239","Type":"ContainerStarted","Data":"408ed0e35c5250a3ea648640596613cfe0a6b20d6985d8110827e69f90944219"} Nov 04 01:48:44.396238 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:44.395869 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-845d896dc4-nvcf6" Nov 04 01:48:44.423783 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:44.423691 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-845d896dc4-nvcf6" podStartSLOduration=1.860838435 podStartE2EDuration="2.423678868s" totalImagesPullingTime="562.840433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:48:44.423088138 +0000 UTC m=+3006.538198258" watchObservedRunningTime="2026-11-04 01:48:44.423678868 +0000 UTC m=+3006.538788986" Nov 04 01:48:44.923329 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:44.923297 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:45.210824 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:45.210742 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:46.138632 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:46.138597 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:46.300418 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:46.300384 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:46.444273 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:46.444188 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:46.581112 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:46.581083 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:47.155175 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:47.155142 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:48.079855 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:48.079811 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:48.745487 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:48.745456 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:48.756436 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:48.756408 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:48:49.424161 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:49.424121 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:50.108893 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:50.108865 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:50.406358 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:50.406287 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-845d896dc4-nvcf6" Nov 04 01:48:50.461733 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:50.461698 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6"] Nov 04 01:48:50.461938 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:50.461913 4852 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" podUID="a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef" containerName="maas-api" containerID="cri-o://c33c6360edd18b094fb008f28eb8e3f514ecbe683bd652482aaaf41a4e7d857e" gracePeriod=30 Nov 04 01:48:50.870818 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:50.870776 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:51.607586 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:51.607538 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:51.717205 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:51.717183 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" Nov 04 01:48:51.856372 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:51.856301 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-openshift-service-ca\") pod \"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef\" (UID: \"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef\") " Nov 04 01:48:51.856372 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:51.856342 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-kube-api-access-vjs6j\" (UniqueName: \"kubernetes.io/projected/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-kube-api-access-vjs6j\") pod \"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef\" (UID: \"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef\") " Nov 04 01:48:51.856628 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:51.856437 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-maas-api-tls\") pod \"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef\" (UID: \"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef\") " Nov 04 01:48:51.856628 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:51.856464 4852 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-metrics-certs\") pod \"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef\" (UID: \"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef\") " Nov 04 01:48:51.856765 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:51.856710 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-openshift-service-ca" pod "a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef" (UID: "a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:48:51.858621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:51.858594 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-metrics-certs" pod "a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef" (UID: "a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:48:51.858621 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:51.858609 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-maas-api-tls" pod "a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef" (UID: "a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:48:51.858750 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:51.858719 4852 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-kube-api-access-vjs6j" pod "a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef" (UID: "a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef"). InnerVolumeSpecName "kube-api-access-vjs6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:48:51.957985 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:51.957950 4852 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-maas-api-tls\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:51.957985 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:51.957977 4852 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-metrics-certs\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:51.957985 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:51.957989 4852 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-openshift-service-ca\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:51.958189 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:51.957999 4852 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjs6j\" (UniqueName: \"kubernetes.io/projected/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef-kube-api-access-vjs6j\") on node \"ip-10-0-140-241.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:52.326431 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:52.326394 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:52.448251 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:52.448212 4852 generic.go:356] "Generic (PLEG): container finished" podID="a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef" containerID="c33c6360edd18b094fb008f28eb8e3f514ecbe683bd652482aaaf41a4e7d857e" exitCode=0 Nov 04 01:48:52.448458 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:52.448333 4852 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" Nov 04 01:48:52.448458 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:52.448348 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" event={"ID":"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef","Type":"ContainerDied","Data":"c33c6360edd18b094fb008f28eb8e3f514ecbe683bd652482aaaf41a4e7d857e"} Nov 04 01:48:52.448458 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:52.448406 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6" event={"ID":"a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef","Type":"ContainerDied","Data":"f4bac244a51faa793cc4fcba4088219e6a0cb95a26be99dfdee6684b13c5ef46"} Nov 04 01:48:52.448458 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:52.448433 4852 scope.go:122] "RemoveContainer" containerID="c33c6360edd18b094fb008f28eb8e3f514ecbe683bd652482aaaf41a4e7d857e" Nov 04 01:48:52.457118 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:52.457102 4852 scope.go:122] "RemoveContainer" containerID="c33c6360edd18b094fb008f28eb8e3f514ecbe683bd652482aaaf41a4e7d857e" Nov 04 01:48:52.457383 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:48:52.457364 4852 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c33c6360edd18b094fb008f28eb8e3f514ecbe683bd652482aaaf41a4e7d857e\": container with ID starting with c33c6360edd18b094fb008f28eb8e3f514ecbe683bd652482aaaf41a4e7d857e not found: ID does not exist" containerID="c33c6360edd18b094fb008f28eb8e3f514ecbe683bd652482aaaf41a4e7d857e" Nov 04 01:48:52.457433 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:52.457392 4852 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33c6360edd18b094fb008f28eb8e3f514ecbe683bd652482aaaf41a4e7d857e"} err="failed to get container status \"c33c6360edd18b094fb008f28eb8e3f514ecbe683bd652482aaaf41a4e7d857e\": rpc error: code = NotFound desc = could not find container \"c33c6360edd18b094fb008f28eb8e3f514ecbe683bd652482aaaf41a4e7d857e\": container with ID starting with c33c6360edd18b094fb008f28eb8e3f514ecbe683bd652482aaaf41a4e7d857e not found: ID does not exist" Nov 04 01:48:52.487098 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:52.487072 4852 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6"] Nov 04 01:48:52.497111 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:52.497078 4852 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-847bf86b74-6mfj6"] Nov 04 01:48:53.112451 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:53.112288 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:53.820090 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:53.820061 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:53.994140 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:53.994110 4852 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef" path="/var/lib/kubelet/pods/a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef/volumes" Nov 04 01:48:54.586842 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:54.586811 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:55.228092 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:55.228059 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:55.828076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:55.828004 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:56.514844 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:56.514811 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:57.170160 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:57.170127 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:57.915121 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:57.915025 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:58.529009 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:58.528969 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:59.235882 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:59.235835 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:48:59.864808 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:48:59.864770 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:48:59.932031 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:48:59.931996 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:00.629676 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:00.629527 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:01.418758 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:01.418724 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:02.116743 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:02.116653 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:02.835272 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:02.835232 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:03.622957 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:03.622922 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:05.356235 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:05.356201 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:05.627926 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:05.627803 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:06.534104 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:06.534072 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:07.407519 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:07.407484 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:08.142006 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:08.141973 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:08.868074 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:08.868042 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:09.538988 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:09.538916 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:10.222179 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:10.222142 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:10.798400 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:10.798364 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:10.977237 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:10.977198 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:11.344200 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:11.344171 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:11.641677 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:11.641591 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:12.019747 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:12.019712 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:12.967745 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:12.967698 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:49:13.321962 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:13.321929 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:13.731954 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:13.731094 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:13.969666 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:13.969635 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:16.848399 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:16.848367 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:17.023052 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:17.023017 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:18.027418 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:18.027390 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:18.192960 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:18.192915 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:18.456886 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:18.456764 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:18.951515 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:18.950185 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:19.722947 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:19.722896 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:20.322998 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:20.322968 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:20.645268 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:20.645192 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:49:21.125688 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:21.125656 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:21.813381 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:21.813347 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:22.499294 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:22.499256 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:23.126985 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:23.126693 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:23.843770 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:23.843730 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:24.506012 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:24.505950 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:25.108259 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:25.108221 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:25.718846 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:25.718813 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:26.408672 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:26.408638 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:27.070091 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:27.070055 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:27.763348 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:27.763313 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:28.415180 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:28.415137 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:29.046829 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:29.046748 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:29.832829 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:29.832779 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:29.866440 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:49:29.866405 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:49:30.522957 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:30.522911 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:31.238492 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:31.238446 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:31.864248 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:31.864212 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:32.031963 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:32.031918 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:32.978427 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:32.978384 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:33.720942 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:33.720894 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:34.433977 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:34.433922 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:35.260456 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:35.260419 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:35.931717 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:35.931686 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:36.623553 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:36.623508 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:37.555826 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:37.555791 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:38.228774 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:38.228723 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:39.008117 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:39.008081 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:39.179771 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:39.179737 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:39.717789 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:39.717755 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:40.334493 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:40.334452 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:41.012787 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:41.012750 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:41.826293 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:41.826256 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:42.530269 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:42.530232 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:43.213423 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:43.213374 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:43.855339 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:43.855290 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:44.530986 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:44.530920 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:45.206581 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:45.206528 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:45.926224 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:45.926193 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:46.611817 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:46.611784 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:47.248461 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:47.248419 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:47.918490 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:47.918450 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:48.620601 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:48.620550 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:49.232220 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:49.232184 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:49.969442 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:49.969408 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:50.640209 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:50.640161 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:51.423412 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:51.423370 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:52.040758 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:52.040723 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:52.717595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:52.717493 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:53.417512 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:53.417478 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:54.059941 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:54.059910 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:54.710107 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:54.710072 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:55.308538 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:55.308498 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:56.046270 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:56.046237 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:56.732327 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:56.732275 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:57.556338 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:57.556306 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:58.254792 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:58.254760 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:58.968473 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:58.968422 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:59.604128 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:49:59.604098 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:49:59.865486 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:49:59.865408 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:50:00.230665 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:00.230553 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:00.918374 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:00.918338 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:01.604994 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:01.599553 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:02.217313 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:02.217264 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:02.939424 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:02.939316 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:03.624399 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:03.624366 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:04.352801 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:04.352742 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:05.148599 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:05.148552 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:05.837278 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:05.837246 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:06.529000 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:06.528933 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:07.246782 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:07.246738 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:07.904457 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:07.904422 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:08.061978 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:08.061942 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:08.843795 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:08.843759 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:09.521247 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:09.521206 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:10.221296 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:10.221110 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:10.606949 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:10.606919 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:10.899404 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:10.899318 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:11.567407 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:11.567322 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:12.216969 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:12.216936 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:12.900812 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:12.900775 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:13.534906 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:13.534867 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:14.201913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:14.201872 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:14.966314 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:14.966280 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:15.612263 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:15.612228 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:16.236810 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:16.236773 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:16.908516 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:16.908484 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:17.617596 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:17.616503 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:18.230359 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:18.230317 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:18.914454 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:18.914414 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:19.628472 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:19.628413 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:20.224232 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:20.224164 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:20.830512 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:20.830475 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:23.405924 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:23.405873 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:23.560114 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:23.560088 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:23.911674 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:23.911643 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:24.651770 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:24.651738 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:25.304275 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:25.304236 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:25.923214 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:25.923102 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:26.567359 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:26.567319 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:27.216091 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:27.216059 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:27.836031 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:27.835971 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:28.517011 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:28.516968 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:29.205707 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:29.205597 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:29.865102 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:50:29.865070 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:50:29.914695 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:29.914659 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:30.604588 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:30.596242 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:31.222287 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:31.222251 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:31.920625 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:31.920583 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:32.652477 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:32.652442 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:33.353136 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:33.353104 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:34.020414 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:34.018950 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:34.755847 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:34.755814 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:35.438076 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:35.438043 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:36.100766 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:36.100727 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:36.725326 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:36.725289 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:37.406411 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:37.406379 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:38.026790 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:38.026760 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:38.719697 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:38.718606 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:39.217395 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:39.217362 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:39.432670 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:39.428771 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:40.114126 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:40.114086 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:40.728868 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:40.728831 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:41.447695 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:41.447664 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:42.121595 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:42.121550 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:43.086726 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:43.086690 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:43.538920 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:43.538871 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:44.248618 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:44.248587 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:44.843470 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:44.843435 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:45.436187 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:45.436133 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:46.132122 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:46.132082 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:46.734417 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:46.734381 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:47.436949 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:47.436911 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:48.217345 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:48.217314 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:48.759883 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:48.759853 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:49.422578 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:49.422511 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:50.110767 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:50.110715 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:50.838752 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:50.838708 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:51.518530 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:51.518490 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:52.170087 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:52.170036 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:52.509335 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:52.509304 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:52.821425 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:52.821341 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:53.526164 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:53.519228 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:54.129929 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:54.129892 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:54.817274 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:54.817235 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:55.425471 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:55.425436 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:56.104556 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:56.104512 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:56.722176 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:56.722147 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:56.961228 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:56.961201 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:57.102973 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:57.102941 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:57.390665 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:57.390582 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:57.660483 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:57.660386 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:57.809948 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:57.809912 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:58.294883 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:58.292790 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:58.817916 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:58.817882 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:58.973609 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:50:58.973509 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:50:59.865622 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:50:59.865591 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:51:00.352148 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:51:00.352119 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:51:04.359028 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:51:04.358978 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:51:04.510482 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:51:04.510448 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:51:04.817808 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:51:04.817771 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:51:05.193112 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:51:05.193022 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:51:07.607457 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:51:07.607421 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:51:13.611635 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:51:13.611593 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:51:29.420726 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:51:29.420694 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:51:29.872404 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:51:29.872362 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:51:32.983293 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:51:32.983256 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:51:33.132021 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:51:33.131983 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:51:34.175710 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:51:34.175658 4852 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-tqg84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:51:34.176369 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:51:34.176336 4852 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqg84" podUID="c13769b4-f82d-460e-aa0f-0e882b40346c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:51:39.262453 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:51:39.262419 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:51:56.818480 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:51:56.818406 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8r2fx"] Nov 04 01:51:59.865843 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:51:59.865803 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:52:09.482778 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:09.482743 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-k8zn6_c322dc68-39b6-40fc-8a6a-ebe9867f1d9c/manager/0.log" Nov 04 01:52:09.631847 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:09.631814 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:52:11.538138 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:11.538109 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-845d896dc4-nvcf6_4f770fbd-3157-4e1d-bb7e-d9aadc559239/maas-api/0.log" Nov 04 01:52:15.669852 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:15.669819 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-8r2fx_8324a6d0-9083-4288-afb5-63729b6430d6/limitador/0.log" Nov 04 01:52:15.853228 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:15.853203 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-b9nwg_742db8d8-0dc9-43b5-a50d-45ab95c45bd3/manager/0.log" Nov 04 01:52:17.172964 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:17.172929 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-c9p2x_f3466cf4-0baf-41e7-ab1a-263c895ab06d/discovery/0.log" Nov 04 01:52:17.355968 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:17.355942 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85448bfccc-jdm4z_3e5369da-d313-4e30-a3b4-cf31dfc43bbf/kube-auth-proxy/0.log" Nov 04 01:52:17.865742 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:17.865711 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-2ttwk_266b7c8e-ffa2-410b-a592-a96f3e034103/istio-proxy/0.log" Nov 04 01:52:18.820643 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:18.820614 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-lkkwt_a232696a-6a25-4803-a2a0-a9888279273c/istio-proxy/0.log" Nov 04 01:52:18.989043 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:18.989011 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-tqg84_c13769b4-f82d-460e-aa0f-0e882b40346c/istio-proxy/0.log" Nov 04 01:52:29.865552 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:52:29.865514 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:52:31.256082 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:31.256048 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:52:31.436865 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:31.436833 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:52:33.196627 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:33.196596 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-k8zn6_c322dc68-39b6-40fc-8a6a-ebe9867f1d9c/manager/0.log" Nov 04 01:52:34.505009 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:34.504976 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:52:34.722114 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:34.722073 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-845d896dc4-nvcf6_4f770fbd-3157-4e1d-bb7e-d9aadc559239/maas-api/0.log" Nov 04 01:52:38.172504 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:38.172472 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-8r2fx_8324a6d0-9083-4288-afb5-63729b6430d6/limitador/0.log" Nov 04 01:52:38.352196 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:38.352165 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-b9nwg_742db8d8-0dc9-43b5-a50d-45ab95c45bd3/manager/0.log" Nov 04 01:52:39.135041 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:39.135004 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-c9p2x_f3466cf4-0baf-41e7-ab1a-263c895ab06d/discovery/0.log" Nov 04 01:52:39.308985 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:39.308951 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:52:39.343431 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:39.343402 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85448bfccc-jdm4z_3e5369da-d313-4e30-a3b4-cf31dfc43bbf/kube-auth-proxy/0.log" Nov 04 01:52:39.883794 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:39.883762 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-2ttwk_266b7c8e-ffa2-410b-a592-a96f3e034103/istio-proxy/0.log" Nov 04 01:52:40.905905 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:40.905874 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-lkkwt_a232696a-6a25-4803-a2a0-a9888279273c/istio-proxy/0.log" Nov 04 01:52:41.106314 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:52:41.106288 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-tqg84_c13769b4-f82d-460e-aa0f-0e882b40346c/istio-proxy/0.log" Nov 04 01:52:59.865586 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:52:59.865529 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:53:29.864893 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:53:29.864810 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:53:38.501267 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:53:38.501235 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 01:53:38.502274 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:53:38.502256 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vst89_c6eada63-a80f-4eff-8076-6f01b9efa23e/ovn-acl-logging/0.log" Nov 04 01:53:39.351312 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:53:39.351276 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:53:49.655899 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:53:49.655868 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:53:49.824243 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:53:49.824208 4852 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" limit=1024 Nov 04 01:53:53.464950 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:53:53.464912 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5v9xx_47fb874a-90bd-41f9-9a8c-0e920d789ca2/global-pull-secret-syncer/0.log" Nov 04 01:53:53.661922 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:53:53.661891 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8d5qr_402ccc5f-d755-4061-a96b-8ddc0495b711/konnectivity-agent/0.log" Nov 04 01:53:53.884287 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:53:53.884256 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-241.us-east-2.compute.internal_cf6706eb85e3c341c82e407ba659da4b/haproxy/0.log" Nov 04 01:53:59.865639 ip-10-0-140-241 kubenswrapper[4852]: E1104 01:53:59.865601 4852 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2w4k" containerName="extractor" Nov 04 01:54:01.795901 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:01.795875 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-8r2fx_8324a6d0-9083-4288-afb5-63729b6430d6/limitador/0.log" Nov 04 01:54:01.865484 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:01.865455 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-b9nwg_742db8d8-0dc9-43b5-a50d-45ab95c45bd3/manager/0.log" Nov 04 01:54:03.039675 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.039636 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dba769a9-3d04-4de1-b3c2-022fd9d98e7e/alertmanager/0.log" Nov 04 01:54:03.064368 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.064341 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dba769a9-3d04-4de1-b3c2-022fd9d98e7e/config-reloader/0.log" Nov 04 01:54:03.102020 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.102003 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dba769a9-3d04-4de1-b3c2-022fd9d98e7e/kube-rbac-proxy-web/0.log" Nov 04 01:54:03.131329 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.131305 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dba769a9-3d04-4de1-b3c2-022fd9d98e7e/kube-rbac-proxy/0.log" Nov 04 01:54:03.155913 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.155892 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dba769a9-3d04-4de1-b3c2-022fd9d98e7e/kube-rbac-proxy-metric/0.log" Nov 04 01:54:03.184143 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.184118 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dba769a9-3d04-4de1-b3c2-022fd9d98e7e/prom-label-proxy/0.log" Nov 04 01:54:03.207711 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.207694 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dba769a9-3d04-4de1-b3c2-022fd9d98e7e/init-config-reloader/0.log" Nov 04 01:54:03.646112 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.646093 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vqnkh_5d50c29d-9104-4dd5-9499-7dd94594666b/node-exporter/0.log" Nov 04 01:54:03.680192 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.680169 4852 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9"] Nov 04 01:54:03.680512 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.680500 4852 cpu_manager.go:414] "containerMap: removing container" podUID="a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef" containerName="maas-api" Nov 04 01:54:03.680559 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.680514 4852 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef" containerName="maas-api" Nov 04 01:54:03.680623 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.680595 4852 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a6c8f1-0a9c-4c04-805b-a8b6ad539fef" containerName="maas-api" Nov 04 01:54:03.683761 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.683739 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9" Nov 04 01:54:03.690217 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.690189 4852 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pplq5\"/\"default-dockercfg-lhzcx\"" Nov 04 01:54:03.692134 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.692105 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9"] Nov 04 01:54:03.693084 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.693064 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pplq5\"/\"kube-root-ca.crt\"" Nov 04 01:54:03.693864 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.693844 4852 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pplq5\"/\"openshift-service-ca.crt\"" Nov 04 01:54:03.694767 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.694752 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vqnkh_5d50c29d-9104-4dd5-9499-7dd94594666b/kube-rbac-proxy/0.log" Nov 04 01:54:03.720992 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.720972 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vqnkh_5d50c29d-9104-4dd5-9499-7dd94594666b/init-textfile/0.log" Nov 04 01:54:03.806371 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.806350 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx42g\" (UniqueName: \"kubernetes.io/projected/807d922a-b5a3-47e8-8885-22dc30feac72-kube-api-access-jx42g\") pod \"perf-node-gather-daemonset-bhvb9\" (UID: \"807d922a-b5a3-47e8-8885-22dc30feac72\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9" Nov 04 01:54:03.806461 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.806375 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/807d922a-b5a3-47e8-8885-22dc30feac72-lib-modules\") pod \"perf-node-gather-daemonset-bhvb9\" (UID: \"807d922a-b5a3-47e8-8885-22dc30feac72\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9" Nov 04 01:54:03.806461 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.806398 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/807d922a-b5a3-47e8-8885-22dc30feac72-proc\") pod \"perf-node-gather-daemonset-bhvb9\" (UID: \"807d922a-b5a3-47e8-8885-22dc30feac72\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9" Nov 04 01:54:03.806591 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.806556 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/807d922a-b5a3-47e8-8885-22dc30feac72-podres\") pod \"perf-node-gather-daemonset-bhvb9\" (UID: \"807d922a-b5a3-47e8-8885-22dc30feac72\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9" Nov 04 01:54:03.806645 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.806629 4852 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/807d922a-b5a3-47e8-8885-22dc30feac72-sys\") pod \"perf-node-gather-daemonset-bhvb9\" (UID: \"807d922a-b5a3-47e8-8885-22dc30feac72\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9" Nov 04 01:54:03.907859 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.907796 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/807d922a-b5a3-47e8-8885-22dc30feac72-sys\") pod \"perf-node-gather-daemonset-bhvb9\" (UID: \"807d922a-b5a3-47e8-8885-22dc30feac72\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9" Nov 04 01:54:03.907859 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.907831 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx42g\" (UniqueName: \"kubernetes.io/projected/807d922a-b5a3-47e8-8885-22dc30feac72-kube-api-access-jx42g\") pod \"perf-node-gather-daemonset-bhvb9\" (UID: \"807d922a-b5a3-47e8-8885-22dc30feac72\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9" Nov 04 01:54:03.907859 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.907850 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/807d922a-b5a3-47e8-8885-22dc30feac72-lib-modules\") pod \"perf-node-gather-daemonset-bhvb9\" (UID: \"807d922a-b5a3-47e8-8885-22dc30feac72\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9" Nov 04 01:54:03.908063 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.907869 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/807d922a-b5a3-47e8-8885-22dc30feac72-proc\") pod \"perf-node-gather-daemonset-bhvb9\" (UID: \"807d922a-b5a3-47e8-8885-22dc30feac72\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9" Nov 04 01:54:03.908063 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.907912 4852 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/807d922a-b5a3-47e8-8885-22dc30feac72-podres\") pod \"perf-node-gather-daemonset-bhvb9\" (UID: \"807d922a-b5a3-47e8-8885-22dc30feac72\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9" Nov 04 01:54:03.908063 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.907921 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/807d922a-b5a3-47e8-8885-22dc30feac72-sys\") pod \"perf-node-gather-daemonset-bhvb9\" (UID: \"807d922a-b5a3-47e8-8885-22dc30feac72\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9" Nov 04 01:54:03.908063 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.907947 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/807d922a-b5a3-47e8-8885-22dc30feac72-lib-modules\") pod \"perf-node-gather-daemonset-bhvb9\" (UID: \"807d922a-b5a3-47e8-8885-22dc30feac72\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9" Nov 04 01:54:03.908063 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.907992 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/807d922a-b5a3-47e8-8885-22dc30feac72-podres\") pod \"perf-node-gather-daemonset-bhvb9\" (UID: \"807d922a-b5a3-47e8-8885-22dc30feac72\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9" Nov 04 01:54:03.908063 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.907993 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/807d922a-b5a3-47e8-8885-22dc30feac72-proc\") pod \"perf-node-gather-daemonset-bhvb9\" (UID: \"807d922a-b5a3-47e8-8885-22dc30feac72\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9" Nov 04 01:54:03.916867 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.916835 4852 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx42g\" (UniqueName: \"kubernetes.io/projected/807d922a-b5a3-47e8-8885-22dc30feac72-kube-api-access-jx42g\") pod \"perf-node-gather-daemonset-bhvb9\" (UID: \"807d922a-b5a3-47e8-8885-22dc30feac72\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9" Nov 04 01:54:03.998162 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:03.998138 4852 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9" Nov 04 01:54:04.169385 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:04.169323 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8685b5f79f-x2l7l_251d4dec-f4d2-4c9d-8734-eab92112714c/telemeter-client/0.log" Nov 04 01:54:04.195021 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:04.194998 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8685b5f79f-x2l7l_251d4dec-f4d2-4c9d-8734-eab92112714c/reload/0.log" Nov 04 01:54:04.228654 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:04.228635 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8685b5f79f-x2l7l_251d4dec-f4d2-4c9d-8734-eab92112714c/kube-rbac-proxy/0.log" Nov 04 01:54:04.267985 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:04.267959 4852 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9"] Nov 04 01:54:04.270903 ip-10-0-140-241 kubenswrapper[4852]: W1104 01:54:04.270882 4852 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod807d922a_b5a3_47e8_8885_22dc30feac72.slice/crio-459a3e5c06a405160824de34b42167c704ada22cc0bb847b069086896fe26000 WatchSource:0}: Error finding container 459a3e5c06a405160824de34b42167c704ada22cc0bb847b069086896fe26000: Status 404 returned error can't find the container with id 459a3e5c06a405160824de34b42167c704ada22cc0bb847b069086896fe26000 Nov 04 01:54:04.825088 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:04.825057 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9" event={"ID":"807d922a-b5a3-47e8-8885-22dc30feac72","Type":"ContainerStarted","Data":"b7dd2b51331afad33ad5d1eeb9017652fecf6397fbb7369401bc678e65036bf0"} Nov 04 01:54:04.825286 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:04.825113 4852 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9" event={"ID":"807d922a-b5a3-47e8-8885-22dc30feac72","Type":"ContainerStarted","Data":"459a3e5c06a405160824de34b42167c704ada22cc0bb847b069086896fe26000"} Nov 04 01:54:04.825286 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:04.825143 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9" Nov 04 01:54:04.843809 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:04.843765 4852 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9" podStartSLOduration=1.8437532970000001 podStartE2EDuration="1.843753297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:54:04.842643827 +0000 UTC m=+3326.957754142" watchObservedRunningTime="2026-11-04 01:54:04.843753297 +0000 UTC m=+3326.958863414" Nov 04 01:54:08.674810 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:08.674779 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z9szb_1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c/dns/0.log" Nov 04 01:54:08.698829 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:08.698805 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z9szb_1a0d5c94-5bd3-4834-9e5e-5ebe04fd979c/kube-rbac-proxy/0.log" Nov 04 01:54:08.725068 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:08.725043 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7vrj7_7cbe8a51-0318-450b-adb2-4abfe1097883/dns-node-resolver/0.log" Nov 04 01:54:09.448606 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:09.448578 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v8q47_565a71a6-1ce7-4078-90fc-f2ccadaa0d17/node-ca/0.log" Nov 04 01:54:10.840465 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:10.840435 4852 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-bhvb9" Nov 04 01:54:11.171462 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:11.171391 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-c9p2x_f3466cf4-0baf-41e7-ab1a-263c895ab06d/discovery/0.log" Nov 04 01:54:11.636127 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:11.636100 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85448bfccc-jdm4z_3e5369da-d313-4e30-a3b4-cf31dfc43bbf/kube-auth-proxy/0.log" Nov 04 01:54:11.726810 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:11.726780 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-2ttwk_266b7c8e-ffa2-410b-a592-a96f3e034103/istio-proxy/0.log" Nov 04 01:54:11.901039 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:11.900974 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-lkkwt_a232696a-6a25-4803-a2a0-a9888279273c/istio-proxy/0.log" Nov 04 01:54:11.932950 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:11.932925 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-tqg84_c13769b4-f82d-460e-aa0f-0e882b40346c/istio-proxy/0.log" Nov 04 01:54:14.449324 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:14.449294 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xw58g_66117eac-ae66-462e-8282-7ef04b047fbe/serve-healthcheck-canary/0.log" Nov 04 01:54:15.172534 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:15.172506 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n2w4k_55e1fd53-92c2-484e-9405-ac5af81ed61b/kube-rbac-proxy/0.log" Nov 04 01:54:15.200392 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:15.200365 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n2w4k_55e1fd53-92c2-484e-9405-ac5af81ed61b/exporter/0.log" Nov 04 01:54:15.227191 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:15.227166 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n2w4k_55e1fd53-92c2-484e-9405-ac5af81ed61b/extractor/0.log" Nov 04 01:54:15.493439 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:15.493361 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-c9p2x_f3466cf4-0baf-41e7-ab1a-263c895ab06d/discovery/0.log" Nov 04 01:54:15.506523 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:15.506458 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85448bfccc-jdm4z_3e5369da-d313-4e30-a3b4-cf31dfc43bbf/kube-auth-proxy/0.log" Nov 04 01:54:15.567533 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:15.567507 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-2ttwk_266b7c8e-ffa2-410b-a592-a96f3e034103/istio-proxy/0.log" Nov 04 01:54:15.671018 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:15.670991 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-lkkwt_a232696a-6a25-4803-a2a0-a9888279273c/istio-proxy/0.log" Nov 04 01:54:15.692272 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:15.692252 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-tqg84_c13769b4-f82d-460e-aa0f-0e882b40346c/istio-proxy/0.log" Nov 04 01:54:17.878989 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:17.878955 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7d9c85cd8c-bscj5_39fe0425-422d-4e5a-a269-04ccb4b6f169/manager/0.log" Nov 04 01:54:17.903855 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:17.903822 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7d9c85cd8c-bscj5_39fe0425-422d-4e5a-a269-04ccb4b6f169/copy-manifests/0.log" Nov 04 01:54:19.163929 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:19.163891 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-c9p2x_f3466cf4-0baf-41e7-ab1a-263c895ab06d/discovery/0.log" Nov 04 01:54:19.184733 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:19.184710 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85448bfccc-jdm4z_3e5369da-d313-4e30-a3b4-cf31dfc43bbf/kube-auth-proxy/0.log" Nov 04 01:54:19.250881 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:19.250850 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-2ttwk_266b7c8e-ffa2-410b-a592-a96f3e034103/istio-proxy/0.log" Nov 04 01:54:19.350046 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:19.350022 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-lkkwt_a232696a-6a25-4803-a2a0-a9888279273c/istio-proxy/0.log" Nov 04 01:54:19.368209 ip-10-0-140-241 kubenswrapper[4852]: I1104 01:54:19.368187 4852 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-tqg84_c13769b4-f82d-460e-aa0f-0e882b40346c/istio-proxy/0.log"